Daily

Get a morning email of failed Databricks jobs

Each morning one email lists every job run that failed overnight, with links to each.

Runs: Daily
Apps: Databricks
AI: Not needed
Access: read jobs

Send this to your assistant

Create a ClickRuntime automation: every day at 7am, list the Databricks job runs from the last 24 hours that failed and email me the job names, error states and links.

Paste it into your assistant. It tells you what it will send or change before it runs live.

Add to ChatGPT Add to Claude

First time? Adding ClickRuntime to your assistant takes about two minutes.

Why use it

Why teams use this automation

Scheduled data jobs fail at night, when nobody is watching. An ingestion job times out, a notebook hits a schema change, a cluster fails to start, and by the time analysts arrive in the morning the dashboards are stale and nobody knows why. The first sign of trouble is a stakeholder asking about wrong numbers.

Each morning, this lists the Databricks job runs from the past day, keeps those whose result state is FAILED, and emails you a list of what failed. You can start fixing problems before downstream users notice them.

Data engineers and analytics leads who run overnight pipelines use it to start the day with the right priorities. The list comes from the job runs that Databricks reports, filtered to a failed result state, so it shows only genuine failures and not cancelled or skipped runs.

What you get out of it

Fix pipelines before dashboards go stale
A failed overnight job means stale reports at 9 a.m. Seeing it at 7 a.m. gives engineers the time to rerun and fix before executives open their dashboards. An overnight failure means stale reports at nine, and learning about it at seven gives engineers time to rerun and fix.
Learn about silent failures
Some failures send no alert because nobody configured one. The morning list covers every job, including ones nobody remembered to monitor. Some failures send no alert because nobody configured one, and the morning list covers every job anyway.
Prioritise by what broke
A concise list helps you decide what to fix first. Critical ingestion jobs go before nice-to-have reports. A concise list helps you decide what to fix first, so critical ingestion jobs come before nice-to-have reports.
Reduce stakeholder surprises
When you can tell users that a job failed and is being fixed, trust holds. Silence followed by correction later does far more harm. Telling users that a job failed and is being fixed keeps their trust, and silence followed by a correction does more harm.
Spot recurring failures
If the same job appears day after day, it needs a real fix, not another rerun. The repeated entries make the pattern impossible to ignore. If the same job appears day after day it needs a real fix, and repetition in the email makes that pattern hard to ignore.

How it works

What happens, step by step

Every morning, ClickRuntime lists the Databricks job runs that finished overnight.
It keeps only the runs that failed.
You get one email with each failed job's name, the error message and a link straight to the run.

Before it runs live

Connect Databricks

What you get

RunsDaily

Get a morning email of failed Databricks jobstoday, 9:00Succeeded

Email · 2 Databricks runs failednightly_orders_etl: FAILED - Task ingest failed with SparkException https://dbc-a1b2c3d4-e5f6.cloud.databricks.com/#job/123/run/456

Build your own

Need it to work differently?

This template is a starting point. Copy the prompt and change it, or describe a different automation from scratch, with your own conditions, more apps or an AI step. Your assistant writes it, tests it and runs it on ClickRuntime.

Your own rules
Conditions, thresholds, filters and exceptions, written the way you'd explain them to a colleague.
Any number of apps
One automation can read from one app and update several others. Over 248 apps are ready, and anything with an API works too.
AI where it helps
Summaries, classification, extraction and drafts use ClickRuntime's built-in AI, with no extra key.
On a schedule or on each event
Runs as often as every 5 minutes, or the moment an app sends an event, and keeps track of what it already handled.

Changes you can ask for

Run it at a different time, more or less often, or only on weekdays
Add an AI step, for example to summarize, sort or draft a reply
Add another app, for example also send it to Slack
Combine it with other Databricks steps into one automation

Under the hood

The code that runs this automation

This is the Python script that runs each time the automation starts. Your AI assistant writes it for you when you send the prompt, and you allow each step in your assistant. To change how the automation works, describe the change to your assistant and it updates the script for you. For every function it can use, see the script reference.

daily_failed_job_report.pyexample

import time
from clickruntime import on_trigger, credentials, http, notify

HOST = "https://dbc-a1b2c3d4-e5f6.cloud.databricks.com"

@on_trigger
def run(event):
    since = int((time.time() - 86400) * 1000)
    data = http.get(f"{HOST}/api/2.1/jobs/runs/list",
                    headers={"Authorization": f"Bearer {credentials.secret('DATABRICKS_TOKEN')}"},
                    params={"start_time_from": since, "completed_only": "true", "limit": 25}).json()
    failed = [r for r in data.get("runs", [])
              if r["state"].get("result_state") in ("FAILED", "TIMEDOUT", "INTERNAL_ERROR")]
    if failed:
        lines = [f"{r['run_name']}: {r['state']['result_state']} - {r['state'].get('state_message', '')[:120]}\n"
                 f"{r['run_page_url']}" for r in failed]
        notify.email_me(f"{len(failed)} Databricks runs failed", "\n\n".join(lines))
    return {"failed": len(failed)}

FAQ

Common questions

What counts as a failed run?

Runs whose result state is FAILED. Cancelled, skipped and timed-out runs have different states and are not listed, though timed-out runs can be added if you want them included. Only runs whose result state is FAILED are listed, and timed-out runs have a different state that can be added on request.

Which time window does it cover?

The previous 24 hours of job runs, so each morning's email covers the night just passed. Tell your AI assistant if you prefer a different window to match when your pipelines run. Each email covers the previous 24 hours of runs, and the window can be shifted to match when your pipelines run.

Does it show the error message?

The list includes the run details returned by the Databricks API. If the run's state message is available, it can be added to each entry, and you would approve that change first. Run details come from the API, and the state message can be added to each entry when it is available, with the change shown first.

What permissions does it need?

read jobs. Nothing else in your accounts is used.

Does it need AI?

No. It runs without AI, so it doesn't use your plan's AI allowance.

How many runs does it use?

One run each time it runs (daily). Checks that find nothing to do and stop right away aren't counted.

Can I change how it works?

Yes. Describe the change to your AI assistant, for example a different schedule, channel or filter. It updates the automation, tests it and tells you what changed.

Daily