Daily

Get a morning email of firing Grafana alerts

Each morning you see every alert rule currently in a firing state, before the stand-up starts.

Runs: Daily
Apps: Grafana Cloud
AI: Not needed
Access: Viewer

Send this to your assistant

Create a ClickRuntime automation: every day at 7:30am email me the Grafana alert rules that are currently firing, with their folder and how long they've been firing.

Paste it into your assistant. It tells you what it will send or change before it runs live.

Add to ChatGPT Add to Claude

First time? Adding ClickRuntime to your assistant takes about two minutes.

Why use it

Why teams use this automation

Grafana alerts fire to contact points, and over a busy week the pages blur together. Some are ones the team has seen and accepted, others quietly keep firing because the on-call person muted the notification and forgot. When a manager asks what is currently broken, nobody can answer without opening the Alerting page and sorting through states. That gap is how known problems turn into permanent background noise. Nobody remembers which ones were accepted.

Each morning the state of your alert rules is read, those currently firing are filtered out, and one email lists them with their labels and how long they have been in that state. It is a snapshot of what needs attention.

It helps on-call engineers and team leads who want a daily check on lingering alerts, separate from the real-time pages.

What you get out of it

Lingering alerts stop being invisible
An alert firing for nine days is probably noise or a real unresolved problem. Either way, listing it forces a decision. An alert firing for nine days is either noise or an unresolved problem, and listing it forces a decision about which one it is.
A shared picture at the morning stand-up
The team starts from the same list of what is firing, without someone sharing a screen to find it. The team starts stand-up from the same list of what is firing, without someone sharing a screen and clicking through the Alerting page to find it.
Noisy rules are identified
A rule that appears in the email every morning is a candidate for a threshold change or removal, which reduces alert fatigue. A rule that appears in the email every morning is a candidate for a threshold change or removal, which reduces alert fatigue over time.
Real issues get a second chance
If a page was missed overnight, the digest catches it before the day is too far gone. If a page was missed overnight or muted by accident, the digest catches it before the day is too far gone and the customer impact grows.
Useful for handovers
The next on-call engineer receives a clear list of what was already burning, so they do not start from zero. The next on-call engineer receives a clear list of what was already burning, so they do not start from zero or repeat investigation work.

How it works

What happens, step by step

Every morning, ClickRuntime checks every Grafana alert rule.
It keeps only the rules that are firing right now and notes how long each has been firing.
You get one email before stand-up listing each firing alert with its folder, name and duration.

Before it runs live

Connect Grafana Cloud

What you get

RunsDaily

Get a morning email of firing Grafana alertstoday, 9:00Succeeded

Email · Firing Grafana alertsPayments / High 5xx rate: firing 6.5h Infra / Disk above 90% on db-1: firing 0.8h

Build your own

Need it to work differently?

This template is a starting point. Copy the prompt and change it, or describe a different automation from scratch, with your own conditions, more apps or an AI step. Your assistant writes it, tests it and runs it on ClickRuntime.

Your own rules
Conditions, thresholds, filters and exceptions, written the way you'd explain them to a colleague.
Any number of apps
One automation can read from one app and update several others. Over 248 apps are ready, and anything with an API works too.
AI where it helps
Summaries, classification, extraction and drafts use ClickRuntime's built-in AI, with no extra key.
On a schedule or on each event
Runs as often as every 5 minutes, or the moment an app sends an event, and keeps track of what it already handled.

Changes you can ask for

Run it at a different time, more or less often, or only on weekdays
Add an AI step, for example to summarize, sort or draft a reply
Add another app, for example also send it to Slack
Combine it with other Grafana Cloud steps into one automation

Under the hood

The code that runs this automation

This is the Python script that runs each time the automation starts. Your AI assistant writes it for you when you send the prompt, and you allow each step in your assistant. To change how the automation works, describe the change to your assistant and it updates the script for you. For every function it can use, see the script reference.

get_a_daily_digest_of_firing_alerts.pyexample

from datetime import datetime, timezone
from clickruntime import on_trigger, credentials, http, notify

GRAFANA = "https://acme.grafana.net"

@on_trigger
def run(event):
    resp = http.get(f"{GRAFANA}/api/prometheus/grafana/api/v1/alerts",
                    headers={"Authorization": f"Bearer {credentials.secret('GRAFANA_TOKEN')}"})
    resp.raise_for_status()
    firing = [a for a in resp.json()["data"]["alerts"] if a["state"].lower() == "alerting"]
    now = datetime.now(timezone.utc)
    lines = []
    for a in firing:
        since = datetime.fromisoformat(a["activeAt"].replace("Z", "+00:00"))
        hours = (now - since).total_seconds() / 3600
        lines.append(f"{a['labels'].get('grafana_folder', '-')} / {a['labels'].get('alertname')}: firing {hours:.1f}h")
    notify.email_me(f"{len(firing)} Grafana alerts firing", "\n".join(lines) or "Nothing is firing.")
    return {"firing": len(firing)}

FAQ

Common questions

Does it include pending alerts?

Only alerts in the firing state are listed. Pending ones, which have not yet met their for duration, are left out so the digest reflects confirmed problems. Only alerts in the firing state are listed. Pending ones, which have not yet met their for duration, are left out so the digest reflects confirmed problems, not blips.

Does it show silenced alerts?

Silenced alerts are normally not firing from the notification point of view, but they can still be in a firing state. A separate weekly list covers silences. Silenced alerts are normally muted from a notification point of view but may still be evaluating as firing, so they can appear. A separate weekly list covers silences.

Will it include alerts from all folders?

By default every rule in the stack is considered. You can limit it to certain folders or label values, such as team=payments, with a specific change. By default every rule in the stack is considered. You can limit it to certain folders or label values, such as team=payments, with a specific change that you approve.

What permissions does it need?

Viewer. Nothing else in your accounts is used.

Does it need AI?

No. It runs without AI, so it doesn't use your plan's AI allowance.

How many runs does it use?

One run each time it runs (daily). Checks that find nothing to do and stop right away aren't counted.

Can I change how it works?

Yes. Describe the change to your AI assistant, for example a different schedule, channel or filter. It updates the automation, tests it and tells you what changed.

Weekly