The original paper helped coding agents reproduce results and ignore missing data
An AI agent checking a published result could use the paper for orientation. It explains the question, expected outputs, and analytical choices. It also hands the agent the number it is meant to verify.
SocSci-Repro-Bench, a 2026 preprint about coding agents and social-science reproduction, measures both effects. Adding the original paper PDF modestly improved overall accuracy and reduced failures. On tasks where data or code were missing, the same PDF made agents less likely to report that reproduction was impossible. Confirmatory wording pushed them further toward producing a plausible answer.
Extra context helped the agents execute the work, but it weakened the independence that makes a reproduction audit useful.
A benchmark where stopping can be correct
The authors built 221 tasks from 54 published papers in political science, sociology, psychology, and communication. The materials cover 13 substantive domains and use R, Python, or Stata. Before inclusion, each reproducible result had to match across three manual code executions.
Ten tasks had no reproducible result because their packages lacked required data or code. The correct response was to state that the result could not be reproduced from the supplied materials. This small subset gives the benchmark an unusual feature: an agent can earn credit by refusing to manufacture an answer.
The public benchmark repository contains task definitions, gold answers, metadata, and analysis scripts. The authors link to their Harvard Dataverse for replication materials.
Two agents ran every task three times in sandboxes without web access or memory between runs: Claude Opus 4.6 through Claude Code and GPT-5.3-Codex through Codex CLI. They could install packages. Their prompts were not identical. According to the paper, Codex needed extra guidance to construct an executable replication script when necessary. The resulting gap is therefore not a model-only comparison.
Claude Code averaged 93.4% task accuracy and 78.0% paper accuracy. Codex averaged 62.1% and 35.8%. A paper counted as reproduced only if the agent answered all of its tasks correctly. Claude Code had no execution failures; Codex failed on 17.8% of tasks and 27.0% of papers.
Those scores apply to these model versions, prompts, sandboxes, and curated runnable materials. The authors caution that the benchmark does not estimate reproducibility across social science.
More context lifted the averages
The researchers next added each original paper PDF to its anonymized replication package. Claude Code's task accuracy rose from 93.4% to 94.5%, while Codex moved from 62.1% to 65.4%. Paper accuracy rose from 78.0% to 80.4% and from 35.8% to 41.4%, respectively.
Codex also got stuck less often. Its failure rate dropped from 17.8% to 12.2% at task level and from 27.0% to 5.6% at paper level. The PDF may have clarified execution order, dependencies, or which output the task requested.
The paper leaked the answer to the audit
In the main evaluation without PDFs, both agents scored 100% on the 10 non-reproducible tasks. After receiving the papers, Claude Code fell to 63.3% and Codex to 90.0%. Instead of reporting that missing materials blocked execution, an agent could extract the expected number from the publication.
The number may match a table in the paper. It still was not reproduced.
A reproduction agent must report what the supplied code and data produce, including when they produce nothing. The PDF is authoritative evidence of what the authors claimed. It is not evidence that the available package can regenerate the claim.
Our article on why agent memory should not turn a preference into evidence examined a related provenance error. SocSci-Repro-Bench shows the same category mistake with a stronger source: an authoritative paper can still be the wrong evidence for a package-level audit.
Polite wording created confirmation pressure
The authors also asked agents to explore "alternative analytically defensible approaches" and select results that "most closely align" with the original paper. That instruction sounds compatible with a robustness check, but it changes the objective. The agent is now searching for a known destination.
Overall task accuracy stayed high or improved: 94.1% for Claude Code and 74.1% for Codex. Codex's task failure rate fell to 0.5%. On the non-reproducible tasks, though, accuracy fell to 70.0% for Claude Code and 60.0% for Codex in this confirmatory condition. The paper says the agents filled gaps with plausible values from the PDF instead of preserving the missing-data verdict.
The pressure to persist helped with broken execution paths. It also made unsupported answers more likely when the required evidence did not exist.
Keep expectations out of the first pass
An agent-assisted reproduction workflow can separate execution help from result verification:
- Give the first run code, data, environment files, and a neutral task. Hide the paper's reported numbers.
- Require a status such as
reproduced,diverged,blocked_missing_data, orblocked_environmentbefore accepting a number. - Record package installations, path rewrites, data substitutions, and specification changes.
- If the paper is revealed in a second phase, use it to investigate discrepancies without replacing the first result.
- Repeat the run with different neutral phrasings. Sensitivity to wording such as "align with" is a risk signal.
This process can wrap any model reached through api.ish.chat, but switching models cannot repair mixed provenance. The evidence channel and the expectation channel need separate records.
SocSci-Repro-Bench is a preprint, its impossible subset contains only 10 tasks, and its tested model versions will age quickly. An agent can improve at finishing assigned work while getting worse at reporting that the evidence is insufficient. A scientific audit has to measure both behaviors.
