Every 5 minutes
Hear when a synthetic monitor starts failing
When one of your New Relic synthetic checks fails, you get one Slack message per monitor until it passes again.
- Runs
- Every 5 minutes
- Apps
- New Relic
- AI
- Not needed
- Access
- user key (read)
Send this to your assistant
Create a ClickRuntime automation: every 5 minutes check my New Relic synthetic monitors and post to #status in Slack when any monitor has a failed check, naming the monitor and location.
First time? Adding ClickRuntime to your assistant takes about two minutes.
Why use it
Why teams use this automation
Synthetic monitors in New Relic visit your login page, run a scripted checkout or ping an API from several locations. They are only valuable if someone notices when they fail. Often the failure sits in New Relic as a red check while the team is busy, and the first they hear is a customer saying the site has been down for an hour. Alert policies exist, but many teams never route them to a channel people read.
Every five minutes ClickRuntime runs an NRQL query on SyntheticCheck for failed results since its last run, and posts to Slack once for each monitor that starts failing, naming the monitor and location. When that monitor passes again, the state resets so the next failure is announced.
Teams that rely on a handful of key user journeys, such as signup and payment, tend to use it.
What you get out of it
Outages are noticed before customers write in
A failing login monitor appears in Slack within five minutes of the failed run. Engineers begin investigating while support is still unaware there is a problem. Engineers begin investigating while support is still unaware that anything is wrong.
One message per failing monitor
Alerting once per monitor prevents a stream of identical posts while the page is down. The channel remains readable, and the thread can be used to coordinate the fix. Alerting once per monitor prevents a stream of identical posts while the page is down, and the thread becomes the place to coordinate.
Location detail helps diagnosis
A failure from only one region suggests a CDN or DNS problem, while a failure from every location suggests your application. Knowing which narrows the search before anyone opens a dashboard.
Key journeys are protected
A scripted browser monitor catches breakages that plain uptime pings miss, such as a payment form that loads but cannot submit. Alerting on those protects the paths that earn revenue.
Recovery closes the loop
When a monitor passes again, the state resets and the next failure produces a new alert. The incident has a visible start and end, which is useful when writing up what happened.
How it works
What happens, step by step
Every five minutes, ClickRuntime asks New Relic for synthetic check results that failed since the last check.
It groups the failures by monitor and location, and counts how many times each failed.
Monitors that are newly failing are posted to #status, for example "Acme login flow failed in Frankfurt, DE (2x)".
A monitor isn't mentioned again until it has passed and then failed once more.
Before it runs live
- Connect New Relic
What you get
Build your own
Need it to work differently?
Your own rules
Conditions, thresholds, filters and exceptions, written the way you'd explain them to a colleague.
Any number of apps
One automation can read from one app and update several others. Over 248 apps are ready, and anything with an API works too.
AI where it helps
Summaries, classification, extraction and drafts use ClickRuntime's built-in AI, with no extra key.
On a schedule or on each event
Runs as often as every 5 minutes, or the moment an app sends an event, and keeps track of what it already handled.
Changes you can ask for
- Run it at a different time, more or less often, or only on weekdays
- Add an AI step, for example to summarize, sort or draft a reply
- Add another app, for example also send it to Slack
- Combine it with other New Relic steps into one automation
Under the hood
The code that runs this automation
from clickruntime import on_trigger, credentials, http, notify
ACCOUNT_ID = 1234567
NRQL = ("SELECT count(*) FROM SyntheticCheck WHERE result = 'FAILED' "
"FACET monitorName, locationLabel SINCE 5 minutes ago")
@on_trigger
def run(event):
query = '{ actor { account(id: %d) { nrql(query: "%s") { results } } } }' % (ACCOUNT_ID, NRQL)
resp = http.post("https://api.newrelic.com/graphql", json={"query": query},
headers={"API-Key": credentials.secret("NEW_RELIC_USER_KEY")})
resp.raise_for_status()
failures = resp.json()["data"]["actor"]["account"]["nrql"]["results"]
if not failures:
return {"failed": 0}
lines = [f"{f['facet'][0]} failed in {f['facet'][1]} ({f['count']}x)" for f in failures]
notify.slack("Synthetic checks failing:\n" + "\n".join(lines), channel="#status")
return {"failed": len(failures)}FAQ
Common questions
Which kinds of monitors does it cover?
Does it alert on a single failed run?
How does it avoid repeating an alert?
What permissions does it need?
Does it need AI?
How many runs does it use?
Can I change how it works?
Related
You might also like
Get a morning list of open alert issues
Each day starts with an email of the New Relic alert issues still open, sorted by priority.
See how it worksGet a weekly performance report for your apps
Every Monday you see throughput, error rate and response time per app, next to the week before.
See how it worksGet a Slack alert when error rate passes 2%
If your app's error rate climbs above your limit, #ops hears once, and again only after it recovers and spikes again.
See how it worksKeep exploring
Set up “Hear when a synthetic monitor starts failing” from your assistant
Copy the prompt above, or connect your assistant first. Free plan, no credit card.