MarketBench found six AI models mispriced their own coding work
An AI agent can fix code well and still answer the assignment question badly: should it get this job? That gap costs money when a router has several workers and must choose which one gets the task.
MarketBench, by Andrey Fradkin and Rohit Krishnan, tests that missing layer. Before each coding attempt, a model quotes its probability of success and expected token use. Those estimates become bids in a simulated procurement auction. The models often solved the work, but their confidence and cost forecasts were too unreliable to allocate it efficiently.
A model's self-report belongs in the routing record. It should not become the routing policy.
Quote the job before doing it
MarketBench uses 93 tasks from SWE-bench Lite, whose issues come from mature open-source Python repositories and are graded by executable tests. Six models evaluated every task, creating 558 model-task rows: Claude Opus 4.5, Claude Sonnet 4.5, Gemini 3 Pro Preview, GPT-5-mini, GPT-5.2, and GPT-5.2-pro.
Before an attempt, each model returned JSON with p_success, estimated_tokens_total, and an optional rationale. The prompt included the issue ID, title, full problem statement, and acceptance commands. Elicitation ran at temperature 0. Real success labels came from a stronger external scaffold with shell access, direct test execution, structured file editing, and multiple rounds of revision.
Most coding scores combine two separate abilities: solving the issue and predicting the outcome before work starts. MarketBench measures both.
Similar results, incompatible forecasts
Realized success for the six models sat in a tight range, from 75.3% to 80.6%. Mean confidence stretched from 61.4% to 92.9%.
Gemini 3 Pro Preview reported 92.9% mean confidence and passed 80.6% of tasks. GPT-5.2 reported 63.2% confidence with the same 80.6% pass rate. GPT-5-mini reported 61.4% confidence and passed 75.3%. Only Claude Opus 4.5 and Claude Sonnet 4.5 achieved positive Brier skill against a naive base-rate forecast.
The cost forecasts were rougher. MarketBench reports a global median estimated-to-actual token ratio of 0.02. Gemini estimated 1,801 tokens on average and used 55,969. Claude Sonnet 4.5 estimated 18,333 and used 53,085. GPT-5.2-pro came closer at 2,647 estimated against 3,970 realized, but the aggregate pattern was severe underestimation.
A router that trusts those quotes can select the boldest bidder rather than the best worker. OmniaBench found a related aggregation problem: near-tied overall scores concealed large differences in what agents could do.
The bold bidder wins too often
MarketBench derives a breakeven bid from token cost, stated success probability, and a $2 failure penalty. It then simulates procurement against reserve prices sampled from $0 to $1.
Every model earned less than its full-information oracle. GPT-5.2 produced about $0.006 in realized profit per task, compared with $0.385 for an oracle that knew which tasks it could solve and abstained on the rest. Gemini won 84.6% of simulated auctions and had the highest realized profit among the tested models. The paper traces that dominance to aggressive bidding despite weak calibration. A bad forecast changed who got the work.
The live scaffold makes the same problem visible, though it does not prove that markets outperform routers. On a common 50-task slice, the market scaffold solved 29 tasks while a solo GPT-5.2 run in the same scaffold solved 24. The market had six workers, however, and could retry with a different model. The difference had a p-value of 0.3. In a matched rerun, a centralized router solved 27 tasks and the market solved 23. The first gain appears to come mainly from model diversity, not the bidding rule.
Past performance improved the quote
The researchers gave each model a held-out summary of its prior behavior. The card listed pass rate, mean stated confidence, typical token underestimation, and repository-specific history when at least five relevant tasks were available.
On the six-model rerun, mean Brier score improved from 0.1835 to 0.1693. Expected calibration error fell from 0.1065 to 0.0616. In the aggregate follow-up, the median estimated-to-actual token ratio moved from 0.1929 to 0.2501. The estimates were still poor, but recorded history pushed them toward reality.
With a self-knowledge prior, the matched market solved 28 of 50 tasks, up from 23 in the matched rerun and one more than the centralized router. The paper describes that result as suggestive. Two nominally identical market conditions had already differed by six tasks, which is a large noise floor for a 50-task study.
Build a calibration ledger
A useful router needs a calibration table for each model, scaffold, repository or task family, and budget tier. Before execution, record predicted success and tokens. After verification, record the result, actual tokens, elapsed time, retries, and execution environment.
Routing can then use measured success on similar tasks, observed cost distributions, and the model's current estimate. Cap that estimate using its calibration history. Abstain when the expected value is negative. Recalibrate whenever the model, prompt, tools, timeout, or scaffold changes.
The Holistic Agent Leaderboard reached a compatible conclusion from broader evaluations: model, scaffold, benchmark, and rollout behavior all affect what an agent score means. EarlyEval showed another side of the same measurement discipline. Saving tokens and measuring capability are separate decisions.
Budget also changes capability. The UK's AI Security Institute found that some cyber tasks were solved only at much larger inference budgets, while cautioning that the result may not extend beyond cyber. MarketBench's codex-direct diagnostic changed both its execution path and time budget. It solved 35 of 50 tasks while using 321.3 million tokens, versus 4.37 million for solo GPT-5.2 and 5.82 million for the market run. This is a diagnostic, not an apples-to-apples ranking.
What MarketBench does not settle
MarketBench covers 93 software-engineering tasks, six model versions, and specific scaffolds. The auction is simulated. Its single-model reserve design is not a competitive marketplace. Token use captures only part of cost, leaving out latency, review, provider limits, and failures whose harms are uneven.
Within that scope, the result is specific and useful. Capable models did not reliably know which jobs they should accept or how much work those jobs would take. Teams routing work through ish.chat, api.ish.chat, or any other agent layer should compare every new self-estimate with verified results from similar jobs. Until the estimates track reality, historical calibration should have the final say.
