ClawBench adds a random-click baseline for browser agents
A browser agent can look competent for a surprisingly long time. It opens the right site, finds a plausible form, fills several fields, and reaches a checkout button. Then it submits the wrong address, forgets an allergy note, or never sends the final request. A screenshot from the middle of that run can still look impressive.
ClawBench treats completion as an observable event rather than a vibe. The open-source benchmark runs agents against live websites, records what they do, blocks consequential final requests, and scores the captured result. Its August 2026 updates also added something less glamorous but unusually clarifying: a random-click baseline.
That baseline does not tell us which model is intelligent. It tells us whether the harness and evaluator award credit for behavior that has no plan at all. Every browser-agent leaderboard should be able to answer that question.
What ClawBench measures
The original ClawBench paper describes 153 tasks across 144 live platforms and 15 life categories. The current repository contains 152 V1 tasks and 129 V2 tasks after two tasks were removed, for 281 tasks across 163 live websites. Tasks include ordering food, booking travel, applying for jobs, managing email, and working with project tools.
These are consequential workflows. Letting an evaluation agent actually place an order or submit an application would be reckless. ClawBench instead watches outgoing browser traffic and intercepts the final request that would commit the action. A run records session video, screenshots, HTTP traffic, browser actions, and agent messages. Human reference runs use the same setup.
The paper's strongest V1 result was a 33.3% success rate. The current project documentation also warns that the paper and shipping corpus differ because two tasks were removed after publication. That detail matters. Benchmark names are not enough to reproduce a number; the task snapshot matters too.
Why random clicking belongs on the leaderboard
ClawBench 0.8.0 added a random-click harness alongside remote-browser CDP support, Gemini-based judging, and an EdgeBench/SForge adapter. Random clicking is a negative control. It should fail, and its failures help expose evaluation bugs.
Imagine a task that asks an agent to add one item to a cart. If the evaluator grants credit whenever any cart endpoint appears in the network log, random activity may eventually stumble into a passing event. A good task definition must distinguish the requested item, quantity, address, options, and final action. The baseline probes whether those distinctions survive the scoring pipeline.
It also tests the environment. A random agent that scores above zero may reveal a task that starts too close to completion, stale browser state, an overbroad URL pattern, or a judge that accepts a plausible payload without enough evidence. None of those failures belong to the model being evaluated.
Compare an agent against a no-skill baseline before comparing it against another agent. A model improvement is hard to interpret when the floor is unknown.
A score combines more than a model
ClawBench leaderboard rows include the model, harness, corpus, pass rate, and wall time. The harness column deserves as much attention as the model column. A harness decides which browser tools are available, how screenshots enter context, what files the agent may read, how retries work, and how the agent is prompted. Changing the harness can change the result even when the model stays fixed.
The scoring documentation makes another separation. For newer V2 runs, stage one asks whether the interceptor captured a request matching the configured endpoint and method. Stage two asks a judge model whether the captured payload fulfills the natural-language instruction. The final result requires both.
The project reports that stage-one-only scores are typically 1.5 to 2 times the final two-stage numbers. Treat that as a finding from ClawBench's own runs, not a universal ratio. It still shows why "reached the endpoint" and "completed the task correctly" should not share one label.
A production browser agent should log at least four separate outcomes:
- navigation reached the intended service;
- the correct action endpoint was reached;
- the payload matched the user's request;
- the action was confirmed or intentionally blocked for review.
Collapsing those states into success hides the difference between progress and completion. The same principle appears in trajectory-based coding-agent evaluation and in CAR-bench's repeated reliability tests, but browser agents add live-site drift and real-world side effects.
Live websites improve realism and weaken comparability
Production websites change. Layouts move, inventory disappears, authentication flows vary, and bot defenses activate. ClawBench deliberately accepts that instability because handling real sites is part of the capability under test. The tradeoff is that two runs on different dates may not face the same environment.
Published traces help. ClawBench exposes task definitions, run metadata, and five layers of execution evidence. Existing traces can be rescored without rerunning the agent, which separates judge changes from browser-run changes. The project also provides a 20-task V1 Lite subset for cheaper iteration.
A leaderboard row is a result for one model, harness, task snapshot, judge configuration, and period of website behavior. It is not a timeless property of the model. A fair comparison must hold those variables steady or state which ones changed.
A practical evaluation recipe
Teams testing a browser agent do not need to begin with all 281 tasks. A smaller internal suite can reveal the most expensive mistakes if it is built carefully.
- Start with the V1 Lite subset or a small set of tasks that matches your product.
- Pin the model identifier, harness version, judge model, rubric, browser runtime, and task commit.
- Run the random-click baseline and one simple scripted baseline before the agent.
- Keep endpoint interception separate from payload correctness and human approval.
- Save traces for failures, then classify site drift, harness errors, judge errors, and model errors separately.
- Repeat the same suite before a release instead of comparing unrelated public leaderboard rows.
If the agent can send email, submit forms, or buy things, use the least-privilege checklist for agent tools alongside the benchmark. Blocking the last irreversible request is a sensible evaluation pattern. It is not a complete production authorization system.
The headline score says that live web work remains difficult. The random-click baseline, harness labels, intercepted requests, payload judge, and downloadable traces explain where a result came from. Those records are what let an engineering team decide whether a change improved the agent or merely changed the test.
