DeepSWE's top three round to 74%. Average cost spans $2.36 to $11.84 per task
The current DeepSWE leaderboard appears to have a winner. GPT-6 Astra at xhigh reasoning scores 74.1%, followed by Gemini 3.8 Flash at high reasoning on 73.8% and Claude Opus 5 at max on 73.6%. Rounded to the whole percentages shown on the page, all three score 74%.
The rest of each row is more useful than that order. Their 95% confidence intervals overlap. Average cost per task runs from $2.36 to $11.84. Average output ranges from roughly 29,600 to 143,200 tokens, while mean agent steps range from 28.8 to 166.3. For a working engineering team, those gaps may matter more than tenths of a percentage point the experiment cannot clearly separate.
New tasks, not mined fixes
DeepSWE has 113 original software-engineering tasks built around 91 active open-source repositories and five languages. Its authors wrote new assignments instead of mining merged GitHub fixes, then kept the reference solutions out of upstream repositories. This reduces the risk that a model encountered an answer during training.
The DeepSWE paper argues that tests shipped with one historical fix are not necessarily fair graders for every possible solution. They can reject a correct alternative or pass an incomplete patch. DeepSWE gives each task a hand-written program verifier that checks observable behavior without comparing the submission to a reference patch.
An independent LLM judge rechecked graded runs and disagreed with DeepSWE's verifiers in 1.4% of cases. Its disagreement rate on SWE-bench Pro's inherited tests was 32.4%. An LLM review is not ground truth, but the contrast makes verifier quality hard to dismiss as benchmark plumbing.
Three leaders, overlapping uncertainty
The live v1.1 data was generated on September 3, 2026. Leading configurations have four whole-benchmark runs, and the published 95% intervals use run-to-run standard error across those repetitions.
- GPT-6 Astra xhigh: 335 passes in 452 attempts, 74.1% with a 2.9-point confidence half-width
- Gemini 3.8 Flash high: 330 passes in 447 attempts, 73.8% with a 1.4-point half-width
- Claude Opus 5 max: 327 passes in 444 attempts, 73.6% with a 3.9-point half-width
Those intervals overlap. GPT-6 Astra occupies the first row in this snapshot, but the data does not establish that it is reliably better at these tasks than the next two configurations.
The unequal attempt counts are also visible. Four complete runs of 113 tasks would yield 452 attempts. Gemini has 447 and Claude has 444. Missing trials should stay separate from incorrect patches because infrastructure failure and task failure answer different questions, even if both can hurt a production service.
Similar pass rates, different operating costs
DeepSWE uses mini-swe-agent for every leaderboard configuration, holding the harness constant. The leading rows still have sharply different operating profiles:
| Configuration | Pass rate | Mean cost/task | Mean output tokens | Mean agent steps |
|---|---|---|---|---|
| GPT-6 Astra xhigh | 74.1% | $6.52 | 29,557 | 28.8 |
| Gemini 3.8 Flash high | 73.8% | $2.36 | 143,243 | 166.3 |
| Claude Opus 5 max | 73.6% | $11.84 | 117,566 | 99.0 |
Gemini is cheapest in the displayed comparison even though it produces the most output tokens and takes the most steps. Model-specific token prices make that possible. A token total alone cannot predict the bill, and neither figure tells you how long a developer waits.
The GPT-6 Astra artifact uses expected launch pricing at all context lengths, with stated rates for uncached input, cache writes, cache reads, output, and compute units. Its $6.52 average is modeled with that price table, not necessarily taken from the historical invoices for those runs. Keep the pricing basis attached when comparing providers.
An average can hide expensive outliers. A configuration may be cheap on routine changes but consume much more on the task your team runs most often. Task-level trajectories are more useful for forecasting than one mean.
Cleaner grading in v1.1
The public repository includes each task's instruction, pinned environment, resource limits, held-out tests, and reference solution. Starting with v1.1, the agent works in one container and the verifier grades in another. The agent commits its changes, a collection hook extracts the patch, and grading applies it in a pristine environment.
This blocks a simple class of false pass. An agent cannot earn credit by changing the tests in its work container or leaving helpful state behind for the grader. Pier, the framework used for the leaderboard, also supplies per-agent network allowlists for tasks where general internet access is disabled.
The tasks, verifier definitions, and leaderboard artifacts are public. A full reproduction still requires access to the models and enough infrastructure, time, and budget to run hundreds of long jobs.
Test the operating profile, not the rank
Choose tasks close to your repositories, languages, and costly failure modes. Run the full suite as context, but report the relevant subset separately. Hold the harness, reasoning setting, model route, environment, timeout, and network policy constant across repeated runs.
Track completed attempts apart from passes, and require a reason for every missing trajectory. Keep pass rate, confidence interval, actual cost, any normalized cost, wall time, output tokens, and agent steps in separate columns. Review a sample of passed and failed patches because a program verifier is stronger than patch matching, but it is not infallible.
When an OpenAI-compatible endpoint such as api.ish.chat sits in the test path, lock and log the served model. Our RuBench analysis shows how an unnoticed fallback can turn a model evaluation into a routing evaluation.
DeepSWE currently gives buyers several configurations with roughly the same measured pass rate. Use that tie. Compare the price, waiting time, output volume, and task-specific failures, then choose the system whose operating profile fits the work.
