What the benchmark actually tests

The official release includes Python, C/C++, Fortran, MATLAB/Octave, and mixed-language projects. The 119 tasks fall into three groups:

• 52 issue-driven tasks begin with a real repository issue or pull request.
• 49 expert-exploratory tasks start from a scientific defect selected through expert analysis.
• 18 engineering-integration tasks require a complete capability chain to work across modules.

The repositories cover chemistry, materials science, biology, biomedical engineering, physics, astronomy, mathematics, and 13 other domains. Chemistry contributes the largest group with 24 tasks. The dataset card also draws a useful operational line: 96 tasks are available in the default unrestricted selection, while 23 require an explicit license opt-in.

Agents receive a frozen repository snapshot, problem statement, required scientific context, and public tests. Private tests and evaluator metadata stay outside the workspace. After the agent submits its patch, the evaluator applies it to a clean baseline. This keeps the grader and hidden checks out of the agent's reach.

The paper tests eight model-and-harness configurations, and the leaders change with the metric. GPT-5.6-sol with Codex records the best private score, Fail2Pass, and regression-preserving Pass2Pass. Claude-Opus-5 with Claude Code has the best overall Pass@1 at 47.90%. DeepSeek-V4-Pro with Claude Code leads the engineering-integration subset at 44.44%.

The model is only one part of the measured system. The harness, reasoning setting, tools, and evaluation path also affect the result. A team routing models through api.ish.chat, for example, should log the model and harness separately instead of folding both into one name.

Most failures are not syntax failures

The authors place unsuccessful repairs into four categories. A scientific knowledge or abstraction deficit means the agent used the wrong mathematical object, physical definition, or domain model. Surface repair describes a patch that improves a visible symptom without tracing it to an independent oracle. Incomplete integration covers local fixes that break data flow, shared invariants, or another part of the system. Failed scientific generalization means the observed case works but an equivalent representation, boundary regime, or unseen condition does not.

Those categories change the questions asked during review. Code clarity and passing tests still matter, but scientific software also carries units, symmetries, conservation laws, coordinate conventions, limiting behavior, and equivalence between representations. A patch can be clean Python, C++, or Fortran and still encode the wrong science.

The U.S. Geological Survey's scientific software policy says software behavior can be part of the scientific record and requires review of scientific veracity, technical quality, and administrative security. Ten simple rules for scientific code review, published in PLOS Computational Biology, recommends starting by running the code and trying to reproduce the result, then reading the implementation with its API and scientific purpose in mind.

More scientific context can help or hurt

The paper also removes auxiliary scientific information from 91 tasks while leaving the repository, executable context, public reproduction entry points, hidden validators, and required minimum context unchanged.

With GPT-5.6-sol, adding that information nudged average public and private scores upward but lowered Pass@1 from 36.26% to 31.87%. Mean token use also fell. With DeepSeek-V4-flash, the same intervention raised Pass@1 from 16.48% to 23.08% and increased token use. The authors say these descriptive differences do not establish statistical significance or a causal effect.

Supplied explanations can narrow the search when they agree with executable evidence. They can also anchor an agent on a diagnosis before it tests other possibilities. Treat a scientific note as evidence that needs checking, not as the answer key.

A safer workflow for scientific repositories

Write the scientific contract before accepting a patch. Record the units, invariants, valid ranges, equivalent representations, and boundary conditions that must survive. If the contract lives only in one researcher's head, no agent can preserve it reliably.

At least one evaluation path should remain independent of the prompt and patch. Depending on the project, that might be an analytic result, a trusted implementation, a conservation check, dimensional analysis, or a separately authored private test. Public tests support iteration, but they should not define the entire acceptance boundary.

The trajectory belongs in the review alongside the diff. The benchmark's failure labels often depend on where the agent looked and why it chose a particular change. BLOGish has guides to grading the full coding-agent trajectory and capturing traces that explain failures. Save the commands, model, harness, tool permissions, token use, and submitted patch.

Risky changes deserve repeated runs. CAR-bench's consistency framing is relevant here because one successful attempt does not show that a workflow is dependable. Disagreement between runs is useful evidence about where a diagnosis is unstable and where human review should concentrate.

A domain expert should own the scientific sign-off. The agent can propose a change and assemble evidence for it. The final decision about a simulation, analysis, or data transformation belongs with someone accountable for the scientific claim built on that software.

SWE-bench Science has limited task counts inside each individual domain, and the authors call their analysis of scientific information preliminary. It does not measure developer productivity or production incident rates. The strongest configuration averaged 96.64% on visible tests yet completed only 47.90% of tasks exactly. Until a team can state and test the scientific invariant independently of a generated patch, that patch is not ready to become part of the instrument.