Every 5 minutes

Hear in Slack when a Fly Machine stops or fails

If a Machine in your app leaves the started state, your channel gets one message naming it and its region.

Runs: Every 5 minutes
Apps: Fly.io
AI: Not needed
Access: org or deploy token (read)

Send this to your assistant

Create a ClickRuntime automation: every 5 minutes, check the Machines in my Fly app 'api-prod' and post to #ops in Slack if any is no longer started. Ignore the Machines in 'preview-web', which autostop.

Paste it into your assistant. It tells you what it will send or change before it runs live.

Add to ChatGPT Add to Claude

First time? Adding ClickRuntime to your assistant takes about two minutes.

Why use it

Why teams use this automation

On Fly.io, a Machine can stop for reasons you did not choose: an out-of-memory kill, a crash loop after a bad deploy, a failed health check, or a host problem. Your app may continue running on other Machines, so the failure is invisible until traffic grows and the remaining ones are overloaded. Many teams find out when a customer complains, hours after the first Machine fell over. Hours of degraded service can follow.

Every five minutes, ClickRuntime lists the Machines in your app, compares each one's state with the state stored last time, and alerts you when a Machine has changed to stopped or failed. It only reports the change, not the continuing condition, so you are not nagged every five minutes.

It suits small production deployments where there is no full monitoring stack yet.

What you get out of it

Learn about crashes within minutes
A Machine that went down at 2 p.m. reaches you at 2:05. Restarting it or rolling back a deploy quickly keeps customers from noticing an outage. Short outages often go unnoticed by users when fixed this quickly.
Avoid overload from lost capacity
If one of three Machines stops, the other two absorb its traffic. Early notice lets you restore capacity before response times degrade for everyone. Capacity is easy to forget until it is gone.
Catch bad deploys faster
A release that makes Machines crash shows up as failed states right after deploy. That is the clearest signal to roll back, ahead of user reports. Rolling back quickly limits the impact on users.
Keep alerts useful
Reporting only changes of state avoids repeated messages for a Machine that stays down. Each alert is new information, so it gets read instead of ignored. That keeps trust in the alerts high.
Add monitoring without a big setup
A full monitoring stack takes time to build. State comparison on a short schedule gives you the most valuable alert on day one. Basic coverage beats none. You can add depth later.

How it works

What happens, step by step

Every five minutes, ClickRuntime lists the Machines in your Fly.io app and whether each is running.
It compares each one with what it saw last time, to spot Machines that have stopped or failed.
When one is no longer running, a message goes to your Slack channel with the Machine's name, ID and region. It isn't repeated while the Machine stays down.

Before it runs live

Connect Fly.io

What you get

RunsEvery 5 minutes

Hear in Slack when a Fly Machine stops or failstoday, 9:00Succeeded

Slack · #ops:red_circle: Fly app api-prod: silent-sun-4821 (148e…, ams) is stopped

Build your own

Need it to work differently?

This template is a starting point. Copy the prompt and change it, or describe a different automation from scratch, with your own conditions, more apps or an AI step. Your assistant writes it, tests it and runs it on ClickRuntime.

Your own rules
Conditions, thresholds, filters and exceptions, written the way you'd explain them to a colleague.
Any number of apps
One automation can read from one app and update several others. Over 248 apps are ready, and anything with an API works too.
AI where it helps
Summaries, classification, extraction and drafts use ClickRuntime's built-in AI, with no extra key.
On a schedule or on each event
Runs as often as every 5 minutes, or the moment an app sends an event, and keeps track of what it already handled.

Changes you can ask for

Act only on the items that match your own rules, and skip the rest
Add an AI step, for example to summarize, sort or draft a reply
Add another app, for example also send it to Slack
Combine it with other Fly.io steps into one automation

Under the hood

The code that runs this automation

This is the Python script that runs each time the automation starts. Your AI assistant writes it for you when you send the prompt, and you allow each step in your assistant. To change how the automation works, describe the change to your assistant and it updates the script for you. For every function it can use, see the script reference.

get_alerted_when_machines_stop_or_crash.pyexample

from clickruntime import on_trigger, credentials, http, notify, state, report_fetched

APP = "api-prod"

@on_trigger
def run(event):
    resp = http.get(f"https://api.machines.dev/v1/apps/{APP}/machines",
                    headers={"Authorization": f"Bearer {credentials.secret('FLY_API_TOKEN')}"})
    resp.raise_for_status()
    down = {m["id"]: m for m in resp.json() if m["state"] != "started"}
    known = set(state.get("down") or [])
    new = [m for mid, m in down.items() if mid not in known]
    report_fetched(len(new))
    if not new:
        if known != set(down):
            state.set("down", list(down))
        return {"new_down": 0}
    lines = [f"{m['name']} ({m['id']}, {m['region']}) is {m['state']}" for m in new]
    notify.slack(f":red_circle: Fly app {APP}:\n" + "\n".join(lines), channel="#ops")
    state.set("down", list(down))
    return {"new_down": len(new)}

FAQ

Common questions

What counts as a failed Machine?

The Fly Machines API reports states such as started, stopped, failed and destroyed. The flow alerts on a move to stopped or failed, and you can add destroyed or suspended if you want those too.

Will it alert me for planned stops?

Machines that you stop on purpose also change state, so they trigger an alert. If you use autostop for cost savings, ask the flow to ignore apps or Machines that are expected to stop. Ignoring lists can be added.

Can it restart the Machine for me?

It can be extended to start a stopped Machine through the API, though restarting a crashing Machine may hide a real fault. Many teams prefer to alert first and add automatic restarts only for known safe cases.

What permissions does it need?

org or deploy token (read). Nothing else in your accounts is used.

Does it need AI?

No. It runs without AI, so it doesn't use your plan's AI allowance.

How many runs does it use?

One run each time it checks. Checks that find nothing new and stop right away aren't counted.

Can I change how it works?

Yes. Describe the change to your AI assistant, for example a different schedule, channel or filter. It updates the automation, tests it and tells you what changed.

Weekly