What SWE-bench Science tests

SWE-bench Science has 119 repository-level tasks drawn from 98 GitHub repositories and 20 scientific domains. Chemistry contributes 24 tasks, followed by materials science and engineering with 16, biology with 13, biomedical engineering with 12, and physics with 11. Six domains appear only once, which limits fine-grained comparisons between fields.

The paper groups the tasks into 52 issue-driven repairs, 49 expert-exploratory investigations, and 18 engineering-integration jobs. The first category starts from a known defect. The second asks the agent to investigate a scientific discrepancy. The third requires changes across files and modules to complete a working capability.

The visible repositories average 80,600 non-empty lines of code and range from 174 to 2,029,051. Reference patches add 117.81 lines on average and delete 44.53. Local clues are rarely the whole problem.

That scale also makes repository navigation part of the evaluation, rather than a setup step that can be ignored.

A high public score can hide an incomplete repair

PublicScore averages the visible test cases. PrivateScore covers hidden cases. Fail2Pass tracks previously failing private tests that the patch repairs, while Pass2Pass checks whether existing behavior survives. Pass@1 is all or nothing: every applicable private test must pass.

For Claude Opus 5 at max reasoning with Claude Code, the paper reports:

• PublicScore: 96.64%
• PrivateScore: 75.11%
• Fail2Pass: 68.60%
• Pass2Pass: 97.37%
• Exact Pass@1: 47.90%

Different systems lead different columns. DeepSeek V4 Pro has a 100% public score and 42.02% exact Pass@1. GPT-5.6-sol leads PrivateScore and Fail2Pass. GLM-5.2 leads Pass2Pass. Claude Opus 5 has the best exact result and leads the issue-driven and expert-exploratory categories, while DeepSeek V4 Pro reaches 44.44% on engineering-integration tasks.

The live leaderboard includes a Hard70 subset selected from the 70 tasks with the lowest average reward across 12 complete model results available on August 28, 2026. Claude Opus 5 leads there at 21.43%, compared with its 47.90% result on all 119 tasks. A deployment estimate needs the task distribution that resembles the deployment.

Four distinct failure patterns

The paper classifies failed repairs four ways. An agent may misunderstand a scientific object, definition, or abstraction. It may patch the visible symptom without finding an independent oracle. A local fix may break interactions elsewhere in the system. Or the repair may work for one observed case but fail on equivalent representations, boundaries, or unseen regimes.

Claude Opus 5 has 58 categorized scientific errors plus four runtime or evaluation-path failures. Only two are classified as surface-level exploration, while 24 involve scientific knowledge or abstraction. DeepSeek V4 Flash records six generalization errors, the lowest count in that category, but has 48 failures involving repair coverage or system integration. The same pass rate would not make those systems operationally equivalent.

Domain guidance helps one model and hurts another

A paired ablation covers 91 tasks. The researchers keep repository evidence, executable context, task objectives, symptoms, input data, and minimal interface documentation. They remove scientific principles, equations, assumptions, expected properties, diagnoses, and scientifically motivated repair strategies.

For GPT-5.6-sol, restoring the scientific material nudges mean public and private scores upward, from 96.70% and 73.23% to 97.80% and 74.06%. Exact Pass@1 drops from 36.26% to 31.87%. The model solves eight tasks only with the material and 12 only without it. Mean input falls from 3.86 million to 3.70 million tokens, and output falls from 43,750 to 40,250.

DeepSeek V4 Flash reacts differently. Scientific information raises Pass@1 from 16.48% to 23.08%. It solves nine tasks only with the material and three only without it. Input rises from 4.84 million to 7.40 million tokens, and output rises from 128,430 to 159,260.

The authors describe these as paired observations, not statistically significant causal effects. Their case review finds that useful context can provide limiting cases, coordinate constraints, independent observables, and authoritative interfaces. A supplied explanation can also anchor an agent on the wrong theory or widen the patch beyond the available evidence.

Build an independent scientific check

Use public tests for diagnosis, then add independently designed checks for invariants, equivalent representations, boundary regimes, and conservation properties absent from the prompt.

Have the agent write down the scientific contract before editing. Each proposed change should point to repository evidence or an executable check. Domain documentation can suggest a hypothesis, but it cannot validate the patch. Grade the result in a clean environment.

The public release publishes separate environment and verifier images pinned by digest and keeps private tests outside the agent workspace. Its default set has 96 unrestricted-license tasks; materializing all 119 requires an explicit decision about the 23 restricted tasks.

Record the model, harness, reasoning setting, task selection, image digests, network policy, and served route. For an OpenAI-compatible endpoint such as api.ish.chat, retain response metadata along with the requested model. Our DeepSWE leaderboard guide explains why score, cost, output, steps, and incomplete attempts should remain separate measurements.

A patch that passes visible tests is ready for independent scientific verification, not automatic acceptance.