Weekly
Get a weekly reminder of muted monitors
Each Monday you see monitors that are muted or under a downtime, so nothing stays silenced by accident.
- Runs
- Weekly
- Apps
- Datadog
- AI
- Not needed
- Access
- monitors_read, monitors_downtime
Send this to your assistant
Create a ClickRuntime automation: every Monday at 9am, email me all Datadog monitors that are muted or covered by a downtime, with when each downtime ends.
First time? Adding ClickRuntime to your assistant takes about two minutes.
Why use it
Why teams use this automation
Muting a Datadog monitor is meant to be temporary. An engineer silences a noisy alert during an incident or a migration, intending to switch it back on later, and then forgets. Weeks later the service has a real problem, and the monitor that should have caught it is still quiet.
Every week, this lists the monitors that are currently muted, along with any scheduled downtimes, and emails you the list. The people responsible are reminded of monitors that were silenced and never restored.
Reliability engineers and team leads use it as a weekly audit, so that silenced alerts do not become permanent blind spots after an incident is over. The list combines currently muted monitors with scheduled downtimes from Datadog, so you see everything that is silent, whatever the way it was silenced.
What you get out of it
Avoid blind spots in production
A muted monitor cannot warn you. A weekly list brings forgotten mutes back into view before a real outage happens behind one. A muted monitor cannot warn you, and a weekly list brings forgotten mutes back into view before an outage hides behind one.
Hold muted monitors accountable
Seeing the list weekly means a mute has to be justified or removed. Temporary silences stop turning into permanent ones. Seeing the list every week means a mute has to be justified or removed, and temporary silences stop becoming permanent.
Clean up expired downtimes
Downtimes that were meant to end but did not still hide alerts. The list helps you find and fix them. Downtimes that were meant to end but did not still hide alerts, and the list helps you find and fix them.
Support incident reviews
After an incident, you can check whether a muted monitor played a part. A history of weekly lists shows what was silent and for how long. After an incident you can check whether a muted monitor played a part, and past lists show what was silent and for how long.
Give leads a simple audit
A manager can scan the list in minutes to confirm that every mute has a reason. No one needs to click through the monitors page. A manager can scan the list in minutes to confirm each mute has a reason, and nobody needs to click through monitors.
How it works
What happens, step by step
Every Monday ClickRuntime finds the Datadog monitors that are muted and the downtimes currently scheduled.
It lists each one with what it covers, when it ends and why, where Datadog has that.
The list arrives in your inbox, so nothing stays silenced by accident.
Before it runs live
- Connect Datadog
What you get
Build your own
Need it to work differently?
Your own rules
Conditions, thresholds, filters and exceptions, written the way you'd explain them to a colleague.
Any number of apps
One automation can read from one app and update several others. Over 248 apps are ready, and anything with an API works too.
AI where it helps
Summaries, classification, extraction and drafts use ClickRuntime's built-in AI, with no extra key.
On a schedule or on each event
Runs as often as every 5 minutes, or the moment an app sends an event, and keeps track of what it already handled.
Changes you can ask for
- Run it at a different time, more or less often, or only on weekdays
- Add an AI step, for example to summarize, sort or draft a reply
- Add another app, for example also send it to Slack
- Combine it with other Datadog steps into one automation
Under the hood
The code that runs this automation
from clickruntime import on_trigger, credentials, http, notify
DD = "https://api.datadoghq.com"
def dd(path, **params):
resp = http.get(f"{DD}{path}", params=params, headers={"DD-API-KEY": credentials.secret("DD_API_KEY"),
"DD-APPLICATION-KEY": credentials.secret("DD_APP_KEY")})
resp.raise_for_status()
return resp.json()
@on_trigger
def run(event):
lines = []
for d in dd("/api/v2/downtime", current_only="true", include="monitor")["data"]:
a = d["attributes"]
target = a.get("monitor_identifier", {})
what = f"monitor {target['monitor_id']}" if "monitor_id" in target else f"tags {', '.join(target.get('monitor_tags', []))}"
ends = (a.get("schedule") or {}).get("end") or "no end date"
lines.append(f"Downtime on {what}: ends {ends}{' - ' + a['message'] if a.get('message') else ''}")
for m in dd("/api/v1/monitor", monitor_tags="", page_size=1000):
if m.get("options", {}).get("silenced"):
lines.append(f"Muted: {m['name']}")
notify.email_me(f"{len(lines)} muted monitors and downtimes", "\n".join(lines) or "Nothing muted.")
return {"count": len(lines)}FAQ
Common questions
Does it include scheduled downtimes?
Can it show who muted a monitor?
How often should I run it?
What permissions does it need?
Does it need AI?
How many runs does it use?
Can I change how it works?
Related
You might also like
Get a morning list of firing Datadog monitors
Every day you see which monitors are in Alert or Warn, without opening Datadog.
See how it worksGet alerted when Datadog usage jumps
If log or host usage climbs well above normal, you hear about it the same day.
See how it worksGet a weekly SLO and error budget report
Every week you see each SLO's status and how much error budget is left, with the ones below target flagged.
See how it worksKeep exploring
Set up “Get a weekly reminder of muted monitors” from your assistant
Copy the prompt above, or connect your assistant first. Free plan, no credit card.