CAR-bench asks what most agent leaderboards avoid: will it work three times?
An AI agent that completes a task once has demonstrated capability. It has not demonstrated reliability.
CAR-bench, an open benchmark presented at ACL 2026 and used for an IJCAI-ECAI 2026 challenge, makes that distinction measurable. Its primary consistency metric, Pass^3, awards a task only when the agent succeeds in all three independent trials. Pass@3, by contrast, asks whether at least one of the three trials worked.
That difference sounds small until a product depends on the result. A support agent that follows policy one time out of three is not partly safe. A vehicle assistant that guesses the destination twice and asks for clarification once is not ready because its best run looked sensible.
The benchmark creates reasons to hesitate
CAR-bench places a tool-using agent inside a synthetic automotive assistant environment. The public benchmark has 254 tasks, 58 interconnected tools, and 19 domain policies. The tools cover navigation, vehicle control, charging, productivity, and related state.
The tasks are split into three categories:
- 100 Base tasks test ordinary multi-turn completion;
- 98 Hallucination tasks remove information or capabilities the request needs;
- 56 Disambiguation tasks give the agent an ambiguous request that requires clarification or additional information gathering.
This design is more revealing than a clean instruction paired with a complete toolset. Real users omit addresses, use vague references, request unavailable actions, and change their minds. A useful agent has to recognize when action is justified and when it should pause.
CAR-bench uses an LLM-simulated user and a synthetic environment, so it is not a direct measurement of production driving safety. Read it as a controlled test of multi-turn behavior, tool use, policy adherence, and limit awareness in one demanding domain, not as proof about every in-car assistant.
Occasional success can hide unstable behavior
The benchmark publishes Pass@3, average single-trial Pass@1, and consistency-focused Pass^3. Consider an agent that solves a task in one trial but fails the other two. It earns credit under Pass@3 and none under Pass^3.
The baseline table shows why both belong in an evaluation report. Claude Opus 4.6 with the benchmark's default scaffold recorded an average Pass^3 of 0.58. Its Base score was 0.80, while Hallucination and Disambiguation were 0.48 and 0.46. GPT-5 recorded an average Pass^3 of 0.54, with 0.66 on Base, 0.60 on Hallucination, and 0.36 on Disambiguation.
Those numbers are benchmark results for named model-and-scaffold configurations, not universal rankings of the models. The environment, policies, prompts, tool schemas, user simulator, and trial count all shape the result. The challenge leaderboard also warns that latency and token telemetry are self-reported and could not be independently verified; its cost estimate applies standardized pricing and a 75% prompt-cache assumption.
The category gap matters more than the model order. Baseline agents that handle ordinary requests reasonably well become much less consistent when they need to admit that a capability is missing or resolve ambiguity before acting.
Completion can conflict with compliance
The CAR-bench authors describe a Completion > Compliance pattern. Agents may prioritize satisfying the request even when the correct behavior is to acknowledge a limit, follow a domain policy, or ask a question.
That behavior is easy to reward accidentally. If an evaluator checks only the final state, a guessed destination can look like decisive action. If it checks only whether a tool was called, an unsupported action can resemble progress. Reliability evaluation needs to inspect why the agent acted and whether the required preconditions were established.
Product teams can separate the evaluation into three graders:
- Did the requested state change happen correctly?
- Did every action comply with policy and permissions?
- Did the agent gather enough information before committing the action?
The tool-interface design guide shows how explicit preconditions and structured failure results make those decisions easier to inspect. The least-privilege checklist covers the enforcement layer that should remain deterministic even when the model makes a bad choice.
Add repeated trials to local agent evaluations
Many internal evaluations run each task once because agent trials cost time and money. A single run is useful while debugging, but it cannot estimate repeatability.
Start by running safety-sensitive and high-value tasks three times with the same configuration. Keep the model version, instructions, tools, repository state, temperature, reasoning settings, and evaluator fixed. Record the complete trajectory for each trial.
Report at least four values:
- single-trial success rate;
- tasks with at least one successful run;
- tasks where every run succeeded;
- policy or permission violations, including violations in otherwise successful runs.
Do not merge those values into one average. A team deciding whether a feature is possible may care about occasional success. A team deciding whether to automate a refund, deploy code, change account access, or control a physical system should care much more about consistent success and the worst failure mode.
The trajectory evaluation guide provides a structure for grading steps, tool calls, and recovery behavior. When comparing models through ISH chat or routing them through ISH API, keep the harness identical and treat the deployed model-plus-agent configuration as the unit under test.
Test the moments when an agent should stop
CAR-bench's automotive setting can be adapted into a general task template. Take a normal workflow and create two variants:
- remove a required capability or piece of information;
- introduce an ambiguity that changes which action is correct.
For a coding agent, remove access to the deployment environment and ask it to deploy. For a support agent, omit the account identifier needed for a refund. For a research agent, provide two entities with the same name. The expected behavior should specify whether the agent must ask, refuse, defer, or use an approved information-gathering tool.
Then run the task repeatedly. Grade fabricated success harshly, even if the answer sounds helpful. A clean refusal can be a passing result when the system lacks the required capability.
CAR-bench does not replace domain-specific testing, human review, or production monitoring. Its repeated-trial metrics separate maximum capability from dependable behavior: the best run shows what an agent can do, while repeated trials show whether a product can rely on it.
