Weekly

Get the week's flaky tests on Friday

Each Friday you get the tests CircleCI has marked as flaky, so someone can fix them.

Runs: Weekly
Apps: CircleCI
AI: Not needed
Access: read

Send this to your assistant

Create a ClickRuntime automation: every Friday at 3pm, email me the flaky tests CircleCI has found in my acme/web-app project, with the job each one runs in.

Paste it into your assistant. It tells you what it will send or change before it runs live.

Add to ChatGPT Add to Claude

First time? Adding ClickRuntime to your assistant takes about two minutes.

Why use it

Why teams use this automation

A flaky test is worse than a failing one. It passes on the second attempt, so nobody fixes it, yet it wastes a few minutes on every unlucky run and teaches developers to ignore red builds. Across a team of twenty, that habit turns into hours of reruns and, eventually, a real regression that gets dismissed as flakiness.

Every week, this asks CircleCI for the tests it has identified as flaky and emails the list. Each entry names the test and how often it flipped between pass and fail, so the people who own the suite can see which ones to repair.

Teams with large end-to-end or integration suites benefit most, because that is where nondeterministic tests accumulate unnoticed. The list comes from CircleCI's own test insights, so it reflects the real pass and fail history on your code rather than a guess from the automation.

What you get out of it

Stop developers dismissing red builds
When a flaky test is known and listed, a failed run still deserves attention. Trust in CI stays high, and an actual bug is less likely to be waved off as another flake.
Reclaim minutes lost on reruns
Each unnecessary rerun can cost ten to twenty minutes of waiting. A weekly list of the offenders lets the team remove them one by one and give that time back across every developer.
Assign owners with a clear list
A named list of tests is easy to hand out in a team meeting. Each person takes one or two, and the next week's email shows whether they dropped off. Ownership removes the bystander effect that lets flaky tests linger for years while everyone assumes somebody else will deal with them.
Expose slow-burning test debt
Flaky tests accumulate slowly. Seeing the list grow from five entries to fifteen is an early signal to pause feature work and spend a sprint on test reliability. A short list is a good sign and a long list is a budget request, so the email doubles as a health indicator for the suite.
Support the case for refactoring
When you ask for time to stabilize a test suite, a dated list of specific tests is far more persuasive than a general complaint that CI feels unreliable. The evidence also makes the cleanup easier to schedule, because you can point at specific tests instead of a vague feeling.

How it works

What happens, step by step

Every Friday, ClickRuntime asks CircleCI which tests it has marked as flaky.
It lists each test with the job it runs in and how many times it flaked.
You get one email with the list, so someone can pick them up and fix them.

Before it runs live

Connect CircleCI

What you get

RunsWeekly

Get the week's flaky tests on Fridaytoday, 9:00Succeeded

Email · Flaky teststest_checkout_applies_coupon (tests.cart) in job unit-tests: flaked 6x

Build your own

Need it to work differently?

This template is a starting point. Copy the prompt and change it, or describe a different automation from scratch, with your own conditions, more apps or an AI step. Your assistant writes it, tests it and runs it on ClickRuntime.

Your own rules
Conditions, thresholds, filters and exceptions, written the way you'd explain them to a colleague.
Any number of apps
One automation can read from one app and update several others. Over 248 apps are ready, and anything with an API works too.
AI where it helps
Summaries, classification, extraction and drafts use ClickRuntime's built-in AI, with no extra key.
On a schedule or on each event
Runs as often as every 5 minutes, or the moment an app sends an event, and keeps track of what it already handled.

Changes you can ask for

Run it at a different time, more or less often, or only on weekdays
Add an AI step, for example to summarize, sort or draft a reply
Add another app, for example also send it to Slack
Combine it with other CircleCI steps into one automation

Under the hood

The code that runs this automation

This is the Python script that runs each time the automation starts. Your AI assistant writes it for you when you send the prompt, and you allow each step in your assistant. To change how the automation works, describe the change to your assistant and it updates the script for you. For every function it can use, see the script reference.

get_alerted_on_flaky_tests.pyexample

from clickruntime import on_trigger, credentials, http, notify

PROJECT = "gh/acme/web-app"

@on_trigger
def run(event):
    resp = http.get(f"https://circleci.com/api/v2/insights/{PROJECT}/flaky-tests",
                    headers={"Circle-Token": credentials.secret("CIRCLECI_TOKEN")})
    resp.raise_for_status()
    tests = sorted(resp.json()["flaky_tests"], key=lambda t: -t["times_flaked"])
    lines = [f"{t['test_name']} ({t['classname']}) in job {t['job_name']}: flaked {t['times_flaked']}x"
             for t in tests]
    notify.email_me(f"{len(tests)} flaky tests in {PROJECT}", "\n".join(lines) or "No flaky tests found.")
    return {"flaky": len(tests)}

FAQ

Common questions

How does CircleCI decide a test is flaky?

CircleCI flags a test as flaky when it has both passed and failed on the same code in recent runs. The email simply reports what CircleCI's test insights return, without applying its own rules on top. The insight is only as good as the data CircleCI receives, so the quality of the list depends on your test reporting setup.

Do I need to upload test results to CircleCI?

Yes. Flaky test detection relies on test results being stored with the store_test_results step in your config. Without that, CircleCI has no per-test data and the list will be empty. Without stored results in the config, there is nothing for CircleCI to analyze and the email will arrive with an empty list.

Can the list go to Slack as well as email?

Yes. Posting the list to a team channel is a small change to the final step. Ask your AI assistant and look over the updated flow before it replaces the weekly email. The change is small and takes effect once you approve it, and the weekly schedule and recipients stay exactly as they were.

What permissions does it need?

read. Nothing else in your accounts is used.

Does it need AI?

No. It runs without AI, so it doesn't use your plan's AI allowance.

How many runs does it use?

One run each time it runs (weekly). Checks that find nothing to do and stop right away aren't counted.

Can I change how it works?

Yes. Describe the change to your AI assistant, for example a different schedule, channel or filter. It updates the automation, tests it and tells you what changed.

Weekly