Skip to content

The moment it happens

Compare answers from two Anthropic models

One question goes to two models you pick, and you get both answers plus a third that reconciles them, by email.

Runs
The moment it happens
Apps
Anthropic API
AI
Not needed
Access
API key

Send this to your assistant

Create a ClickRuntime automation: give me a link I can post questions to. Send each question to two Anthropic models using my Anthropic API key, have the larger model reconcile the two answers, and email me all three with the token cost of each.

Paste it into your assistant. It tells you what it will send or change before it runs live.

First time? Adding ClickRuntime to your assistant takes about two minutes.

Why use it

Why teams use this automation

Different Claude models suit different jobs, and choosing between them is often a guess. A founder deciding whether the cheaper model is good enough for customer replies, or a developer testing a prompt, ends up pasting the same question into several tools and comparing answers by eye. For important questions, a single answer is also a single point of failure: it may be confident and wrong, and there is nothing to check it against. Nobody records how much each answer cost, so the cheaper model never gets a fair trial.

When a question arrives, the automation calls the Messages API once for each model you pick, for example Sonnet and Opus. It can then make a judge call that merges the answers or checks them for disagreement, and it emails the results with the token costs of each call.

Developers testing prompts, teams choosing a model for production, and anyone who wants a second opinion on a high-stakes answer can use it.

What you get out of it

  • Model choice is based on evidence

    Seeing two answers side by side, with their costs, shows whether the cheaper model is good enough. You choose with data, not on reputation, and may save real money at scale.

  • Disagreements reveal weak answers

    When two models give different answers, that is a signal to look more closely. A merged or judged result catches errors that a single answer would have hidden. Disagreement is cheap to detect and expensive to miss.

  • Costs are visible per call

    Token counts for each model come with the results, so you can calculate the price of a thousand questions before you move a prompt into production. A cheaper model may do the job for a tenth of the price.

  • Prompts can be tested quickly

    Sending the same prompt to several models shows which handles your wording best, and shortens the time spent rewriting instructions by trial and error. A few runs often show which prompt style a model prefers, so you refine the instructions with real results.

  • Important answers get a second look

    For decisions with real consequences, such as a contract summary or a customer reply on a refund, a second model's check is cheap compared with the cost of a wrong answer.

How it works

What happens, step by step

  1. When you send a question to your personal ClickRuntime link, the automation starts straight away.

  2. The question goes to two Claude models, a fast one and a stronger one, and each writes its own answer.

  3. A third, most capable Claude model reads both answers, combines them into one best answer and notes where they disagree.

  4. You get one email with all three answers and how many tokens each one used.

Before it runs live

  • Connect Anthropic API
  • Paste your automation's link into the other app's webhook settings

What you get

RunsThe moment it happens
Compare answers from two Anthropic modelsjust nowSucceeded
Email · Claude answers== claude-opus-5-5 (912 in / 340 out tokens) == Both answers agree on the main steps; B adds the rate-limit caveat.

Build your own

Need it to work differently?

This template is a starting point. Copy the prompt and change it, or describe a different automation from scratch, with your own conditions, more apps or an AI step. Your assistant writes it, tests it and runs it on ClickRuntime.
  • Your own rules

    Conditions, thresholds, filters and exceptions, written the way you'd explain them to a colleague.

  • Any number of apps

    One automation can read from one app and update several others. Over 248 apps are ready, and anything with an API works too.

  • AI where it helps

    Summaries, classification, extraction and drafts use ClickRuntime's built-in AI, with no extra key.

  • On a schedule or on each event

    Runs as often as every 5 minutes, or the moment an app sends an event, and keeps track of what it already handled.

Changes you can ask for

  • Act only on the items that match your own rules, and skip the rest
  • Add an AI step, for example to summarize, sort or draft a reply
  • Add another app, for example also send it to Slack
  • Combine it with other Anthropic API steps into one automation

Under the hood

The code that runs this automation

This is the Python script that runs each time the automation starts. Your AI assistant writes it for you when you send the prompt, and you allow each step in your assistant. To change how the automation works, describe the change to your assistant and it updates the script for you. For every function it can use, see the script reference.
compare_or_combine_answers_from_specific_claude_models.pyexample
from clickruntime import on_event, credentials, notify, http

def claude(model, prompt, **extra):
    resp = http.post("https://api.anthropic.com/v1/messages",
                     headers={"x-api-key": credentials.secret("ANTHROPIC_API_KEY"), "anthropic-version": "2023-06-01"},
                     json={"model": model, "max_tokens": 1024, "messages": [{"role": "user", "content": prompt}], **extra},
                     timeout=120)
    resp.raise_for_status()
    return resp.json()

def text(msg):
    return "".join(b["text"] for b in msg["content"] if b["type"] == "text")

@on_event
def handle(event):
    question = (event.json or {}).get("question") or event.body
    a = claude("claude-haiku-5-5", question)
    b = claude("claude-sonnet-5-5", question)
    judge = claude("claude-opus-5-5", f"Question: {question}\n\nAnswer A:\n{text(a)}\n\nAnswer B:\n{text(b)}\n\n"
                   "Reconcile these into one best answer and note where they disagree.")
    parts = [f"== {m['model']} ({m['usage']['input_tokens']} in / {m['usage']['output_tokens']} out tokens) ==\n{text(m)}"
             for m in (a, b, judge)]
    notify.email_me(f"Claude answers: {question[:80]}", "\n\n".join(parts))
    return {"models": [m["model"] for m in (a, b, judge)]}

FAQ

Common questions

Which Claude models can I compare?
Any model available through the Anthropic API, such as Sonnet, Opus or Haiku. List the model IDs you want, and the automation calls each in turn with the same prompt. Check Anthropic's documentation for the current list and exact IDs.
What does the judge call do?
It receives the question and the answers, and is asked to merge them into one answer or to say where they disagree and which is better supported. You can choose either behavior. Skip the judge if you only want raw comparison.
Is it much more expensive than asking one model?
It costs the sum of each call, plus the judge call if you use it. The email shows token counts for each, so you can see the total before you scale it up. For important questions the extra cost is usually small.
What permissions does it need?
API key. Nothing else in your accounts is used.
Does it need AI?
No. It runs without AI, so it doesn't use your plan's AI allowance.
How many runs does it use?
One run for each event the app sends.
Can I change how it works?
Yes. Describe the change to your AI assistant, for example a different schedule, channel or filter. It updates the automation, tests it and tells you what changed.

Set up “Compare answers from two Anthropic models” from your assistant

Copy the prompt above, or connect your assistant first. Free plan, no credit card.