Daily

Run a daily scrape and get the results by email

Your scraper runs every morning and you get the record count and download link once the snapshot is ready.

Runs: Daily
Apps: Bright Data
AI: Not needed
Access: API key

Send this to your assistant

Create a ClickRuntime automation: every day at 6am, trigger my Bright Data scraper for these 4 competitor URLs, and when the snapshot is ready, email me how many records it has and the download link.

Paste it into your assistant. It tells you what it will send or change before it runs live.

Add to ChatGPT Add to Claude

First time? Adding ClickRuntime to your assistant takes about two minutes.

Why use it

Why teams use this automation

A product or pricing manager who wants to know what competitors charge usually starts with someone checking websites by hand. It works for a week, then becomes a task that is skipped when things get busy, and when a rival drops prices by 15 percent, the team finds out from a lost deal. Writing and maintaining scrapers in-house is its own project, with sites changing layout and blocking requests.

Each day, ClickRuntime starts a Bright Data dataset collection for your list of competitor pages and gives Bright Data a notify address. When the collection finishes, Bright Data calls that address, the workflow fetches the snapshot details and emails you that the data is ready, with a link. You do not poll and you are not told until there is something to look at.

Teams in e-commerce, travel and marketplaces, where prices and listings move constantly, tend to use this.

What you get out of it

Competitor moves are seen the same day
A fresh snapshot every morning means a price cut or a new product is noticed within hours, not at the quarterly review. You can respond while the change still matters.
No scrapers to build or repair
Bright Data handles the collection and the blocking, so your team does not maintain brittle code. Engineers stay on the product while the market data arrives on a schedule. That saves repeated manual checking of a job status page.
You are told only when data is ready
Collections can take minutes or longer. The notify callback means the email arrives when the snapshot is complete, so you never open an empty result. Running the same collection daily builds a time series of competitor data, and trends in discounting, stock levels and new listings become visible, which no one-off check could ever show.
Consistent history for analysis
Running the same collection daily builds a time series of competitor data. Trends in discounting and stock become visible, which no one-off check could show. Pricing or positioning arguments can be settled with a dated snapshot, and leaders see what competitors actually offered on a given day rather than someone's memory of a webpage they glanced at.
Decisions rest on evidence
Pricing or positioning arguments can be settled with a dated snapshot. Leaders see what competitors actually offered on a given day rather than someone's memory of a webpage. Reviewing the data becomes a habit tied to a single email.

How it works

What happens, step by step

Every morning, ClickRuntime starts a collection run for your Bright Data dataset.
When Bright Data reports that the snapshot is ready, ClickRuntime looks up how many records it holds.
You get an email with the record count and a link to download the results as a CSV file.

Before it runs live

Connect Bright Data

What you get

RunsDaily

Run a daily scrape and get the results by emailtoday, 9:00Succeeded

Email · Competitor scrape ready1,284 records in snapshot s_m2k9x1. Download: https://api.brightdata.com/datasets/v3/snapshot/s_m2k9x1?format=csv

Build your own

Need it to work differently?

This template is a starting point. Copy the prompt and change it, or describe a different automation from scratch, with your own conditions, more apps or an AI step. Your assistant writes it, tests it and runs it on ClickRuntime.

Your own rules
Conditions, thresholds, filters and exceptions, written the way you'd explain them to a colleague.
Any number of apps
One automation can read from one app and update several others. Over 248 apps are ready, and anything with an API works too.
AI where it helps
Summaries, classification, extraction and drafts use ClickRuntime's built-in AI, with no extra key.
On a schedule or on each event
Runs as often as every 5 minutes, or the moment an app sends an event, and keeps track of what it already handled.

Changes you can ask for

Run it at a different time, more or less often, or only on weekdays
Add an AI step, for example to summarize, sort or draft a reply
Add another app, for example also send it to Slack
Combine it with other Bright Data steps into one automation

Under the hood

The code that runs this automation

This is the Python script that runs each time the automation starts. Your AI assistant writes it for you when you send the prompt, and you allow each step in your assistant. To change how the automation works, describe the change to your assistant and it updates the script for you. For every function it can use, see the script reference.

scheduled_dataset_scrape_with_result_email.pyexample

from clickruntime import on_trigger, on_event, credentials, http, notify

DATASET_ID = "gd_l7q7dkf244hwjntr0"
TRIGGER_LINK = "https://hooks.clickruntime.com/your-trigger-link"  # this automation's trigger link
URLS = ["https://northwind.example/products", "https://acme.example/shop",
        "https://contoso.example/catalog", "https://fabrikam.example/store"]

def headers():
    return {"Authorization": f"Bearer {credentials.secret('BRIGHTDATA_API_KEY')}"}

@on_trigger
def run(event):
    resp = http.post("https://api.brightdata.com/datasets/v3/trigger", headers=headers(),
                     params={"dataset_id": DATASET_ID, "notify": TRIGGER_LINK},
                     json=[{"url": u} for u in URLS])
    resp.raise_for_status()
    return {"snapshot_id": resp.json()["snapshot_id"]}

@on_event
def ready(event):
    snap = (event.json or {}).get("snapshot_id")
    progress = http.get(f"https://api.brightdata.com/datasets/v3/progress/{snap}", headers=headers()).json()
    if progress.get("status") != "ready":
        return {"status": progress.get("status")}
    link = f"https://api.brightdata.com/datasets/v3/snapshot/{snap}?format=csv"
    notify.email_me("Competitor scrape ready",
                    f"{progress.get('records', 0)} records in snapshot {snap}.\nDownload: {link}")
    return {"records": progress.get("records")}

FAQ

Common questions

How do I know when the scrape is finished?

Bright Data sends a notification to the address supplied when the collection starts. That triggers the second part of the workflow, which fetches the snapshot information and emails you. It saves you from checking repeatedly or guessing at how long a collection takes.

Does the email contain the scraped data itself?

The email tells you the data is ready and links to the snapshot, so large datasets are not pasted into a message. The snapshot can be downloaded from Bright Data. A summary step can be added if you want key figures in the email.

Which Bright Data account setup is needed?

You need a Bright Data account with access to the dataset product you intend to use, plus an API key, which ClickRuntime keeps as a secret. Bright Data's own usage charges apply to the collections that run.

What permissions does it need?

API key. Nothing else in your accounts is used.

Does it need AI?

No. It runs without AI, so it doesn't use your plan's AI allowance.

How many runs does it use?

One run each time it runs (daily). Checks that find nothing to do and stop right away aren't counted.

Can I change how it works?

Yes. Describe the change to your AI assistant, for example a different schedule, channel or filter. It updates the automation, tests it and tells you what changed.

On a schedule