Daily

Track flaky CI jobs in GitHub Actions

Each morning Slack lists the GitHub Actions workflows that failed and then passed on a re-run in the last week, so you know which tests are flaky.

Runs: Daily
Apps: GitHub, Slack
AI: Not needed
Access: actions read; chat:write

Send this to your assistant

Create a ClickRuntime automation: every morning, look at the last 7 days of GitHub Actions runs in acme/app and post to #eng the workflows that failed and then passed on retry, with how often.

Paste it into your assistant. It tells you what it will send or change before it runs live.

Add to ChatGPT Add to Claude

First time? Adding ClickRuntime to your assistant takes about two minutes.

Why use it

Why teams use this automation

A team's test suite fails a few times a week for no clear reason. Developers have learned to click re-run until it goes green, which wastes time on every pull request and teaches everyone to ignore red builds, including the real ones.

A daily list of which workflows needed a retry to pass, and how often, turns a vague annoyance into specific targets. The team can fix the worst offenders first and restore trust in a red build.

What you get out of it

Flaky tests get named
The list shows exactly which workflows fail and then pass. Vague complaints become specific work items.
Worst offenders first
Workflows are ranked by how many retries they needed. Fixing the top one or two often removes most of the pain.
Red builds mean something again
When flakiness is fixed, a failing build points to a real problem. Developers stop reflexively clicking re-run.
Wasted time becomes visible
Seeing the retry count each day shows how much time flaky jobs cost. That makes the case for fixing them.

How it works

What happens, step by step

Every morning, ClickRuntime reads the last 7 days of GitHub Actions runs for your repository.
It finds runs that only passed after being re-run on the same commit.
It counts these retries for each workflow and ranks them.
The list is posted to your #eng Slack channel.

Before it runs live

Connect GitHub
Connect Slack, then invite the app to the channel (type /invite @ClickRuntime there)

What you get

RunsDaily

Track flaky CI jobs in GitHub Actionstoday, 9:00Succeeded

Slack · #engFlaky workflows in acme/app, last 7 days • e2e-checkout: passed on retry 9 times out of 64 runs

Build your own

Need it to work differently?

This template is a starting point. Copy the prompt and change it, or describe a different automation from scratch, with your own conditions, more apps or an AI step. Your assistant writes it, tests it and runs it on ClickRuntime.

Your own rules
Conditions, thresholds, filters and exceptions, written the way you'd explain them to a colleague.
Any number of apps
One automation can read from one app and update several others. Over 248 apps are ready, and anything with an API works too.
AI where it helps
Summaries, classification, extraction and drafts use ClickRuntime's built-in AI, with no extra key.
On a schedule or on each event
Runs as often as every 5 minutes, or the moment an app sends an event, and keeps track of what it already handled.

Changes you can ask for

Run it at a different time, more or less often, or only on weekdays
Add an AI step, for example to summarize, sort or draft a reply
Add another app, for example also send it to Google Sheets
Combine it with other GitHub and Slack steps into one automation

Under the hood

The code that runs this automation

This is the Python script that runs each time the automation starts. Your AI assistant writes it for you when you send the prompt, and you allow each step in your assistant. To change how the automation works, describe the change to your assistant and it updates the script for you. For every function it can use, see the script reference.

flaky_ci_job_tracker.pyexample

from collections import Counter
from datetime import date, timedelta
from clickruntime import on_trigger, credentials, http, notify

def gh(method, path, **kw):
    resp = http.request(method, f"https://api.github.com{path}", headers={
        "Authorization": f"Bearer {credentials.connection('github', 'work')}", "Accept": "application/vnd.github+json"}, **kw)
    return resp.raise_for_status() or (resp.json() if resp.content else None)

REPO = "acme/app"

@on_trigger
def run(event):
    since = date.today() - timedelta(days=7)
    runs = gh("GET", f"/repos/{REPO}/actions/runs", params={"created": f">={since}", "status": "completed", "per_page": 100})["workflow_runs"]
    flaky = Counter(r["name"] for r in runs if r["run_attempt"] > 1 and r["conclusion"] == "success")
    total = Counter(r["name"] for r in runs)
    if not flaky:
        return {"flaky": 0}
    lines = [f"• {name}: passed on retry {n} times out of {total[name]} runs" for name, n in flaky.most_common()]
    notify.slack(f"Flaky workflows in {REPO}, last 7 days\n" + "\n".join(lines), channel="#eng")
    return {"flaky": dict(flaky)}

FAQ

Common questions

How does it decide a job is flaky?

A run that failed on its first attempt and passed on a later attempt of the same commit is counted as flaky. Runs that failed and stayed failed are not counted.

Can it cover several repositories?

Yes. List the repositories and the report groups flaky workflows by repository.

Does it post when nothing is flaky?

No. If no workflow needed a retry in the last week, nothing is posted.

What permissions does it need?

actions read; chat:write. Nothing else in your accounts is used.

Does it need AI?

No. It runs without AI, so it doesn't use your plan's AI allowance.

How many runs does it use?

One run each time it runs (daily). Checks that find nothing to do and stop right away aren't counted.

Can I change how it works?

Yes. Describe the change to your AI assistant, for example a different schedule, channel or filter. It updates the automation, tests it and tells you what changed.

On a schedule