On a schedule
Get a Slack alert when a job runs too long
Any run still going after two hours is flagged in Slack once, so stuck jobs don't burn compute all night.
- Runs
- On a schedule
- Apps
- Databricks
- AI
- Not needed
- Access
- read jobs
Send this to your assistant
Create a ClickRuntime automation: every 15 minutes, check active Databricks job runs and post to #data-eng in Slack if any have been running for more than 2 hours. Only alert once per run.
First time? Adding ClickRuntime to your assistant takes about two minutes.
Why use it
Why teams use this automation
A Databricks job that should take twenty minutes and is still running after two hours is almost always in trouble. A skewed join is stuck on one partition, a cluster is waiting for resources, or a notebook is looping. Every extra hour burns compute and holds up whatever depends on the result.
On a schedule, this lists the active runs, works out how long each has been running, and sends a Slack message the first time a run passes two hours. The alert is sent once per run, so the channel does not fill with repeated warnings.
Platform teams and engineers who own critical pipelines use it to catch hangs before the cost and the delays pile up. Duration is worked out from the start time of each active run, so the alert measures what is happening now, and it is sent once per run so the channel stays readable.
What you get out of it
Stop hung jobs wasting compute
A stuck job costs money by the hour. Hearing about it at the two-hour mark means you can cancel it before it burns an entire night of compute. A stuck job costs money by the hour, and hearing about it at two hours means you can cancel before a night of compute is wasted.
Protect downstream deadlines
Reports and models waiting on a slow job miss their deadlines. An early warning gives the owner time to rerun or notify stakeholders. Reports and models waiting on a slow job miss deadlines, and an early warning gives the owner time to rerun or tell stakeholders.
Alert once per run
Because each run is reported a single time, you can trust the channel. A repeated warning every poll would soon be muted and ignored. Each run is reported a single time, so the channel stays trustworthy, and a repeated warning every poll would soon be muted.
Find jobs that are slowly degrading
Jobs that regularly cross the threshold show growing data volumes or inefficient code. Those are the ones to optimise next. Jobs that regularly cross the threshold reveal growing data volumes or inefficient code, which are the ones to optimize next.
Give on-call engineers clear information
The message names the run, so the person on call knows exactly what to investigate. They do not have to scan the jobs page for suspicious entries. The message names the run, so the on-call engineer knows exactly what to investigate and does not scan the jobs page.
How it works
What happens, step by step
Every 15 minutes, ClickRuntime checks which Databricks jobs are currently running.
It works out how long each one has been going.
Any run that has passed two hours is posted to your Slack channel with its name, how long it has run and a link.
Each run is flagged only once, however long it keeps going.
Before it runs live
- Connect Databricks
What you get
Build your own
Need it to work differently?
Your own rules
Conditions, thresholds, filters and exceptions, written the way you'd explain them to a colleague.
Any number of apps
One automation can read from one app and update several others. Over 248 apps are ready, and anything with an API works too.
AI where it helps
Summaries, classification, extraction and drafts use ClickRuntime's built-in AI, with no extra key.
On a schedule or on each event
Runs as often as every 5 minutes, or the moment an app sends an event, and keeps track of what it already handled.
Changes you can ask for
- Act only on the items that match your own rules, and skip the rest
- Add an AI step, for example to summarize, sort or draft a reply
- Add another app, for example also send it to Slack
- Combine it with other Databricks steps into one automation
Under the hood
The code that runs this automation
import time
from clickruntime import on_trigger, credentials, http, state, notify
HOST = "https://dbc-a1b2c3d4-e5f6.cloud.databricks.com"
MAX_HOURS = 2
@on_trigger
def run(event):
data = http.get(f"{HOST}/api/2.1/jobs/runs/list",
headers={"Authorization": f"Bearer {credentials.secret('DATABRICKS_TOKEN')}"},
params={"active_only": "true", "limit": 25}).json()
alerted = set(state.get("alerted") or [])
active = data.get("runs", [])
for r in active:
hours = (time.time() * 1000 - r["start_time"]) / 3_600_000
if hours > MAX_HOURS and r["run_id"] not in alerted:
notify.slack(f"{r['run_name']} has been running for {hours:.1f}h\n{r['run_page_url']}",
channel="#data-eng")
alerted.add(r["run_id"])
state.set("alerted", [i for i in alerted if i in {r["run_id"] for r in active}])
return {"active": len(active)}FAQ
Common questions
Can I change the two hour limit?
How often does it check for long runs?
Does it cancel the run automatically?
What permissions does it need?
Does it need AI?
How many runs does it use?
Can I change how it works?
Related
You might also like
Get a morning email of failed Databricks jobs
Each morning one email lists every job run that failed overnight, with links to each.
See how it worksGet an alert when daily DBU spend goes over budget
If yesterday's DBU usage passed your budget, you know first thing in the morning.
See how it worksSee new job listings as they're posted
Jobs that match your keywords show up in a Slack channel, each one only once.
See how it worksKeep exploring
Set up “Get a Slack alert when a job runs too long” from your assistant
Copy the prompt above, or connect your assistant first. Free plan, no credit card.