SocSci-Repro-Bench found a reproduction agent can be accurate and still too eager to agree
A coding agent looks well suited to reproducing a published result. It can inspect the replication package, repair the environment, run the analysis, and extract the requested number. The paper tells it what that number ought to be.
But reproduction is meant to test whether those materials lead to that number. Once the published answer is visible, an agent can mistake agreement for evidence.
SocSci-Repro-Bench caught that mistake. It contains 221 tasks drawn from 54 social science papers in political science, sociology, psychology, and communication. Claude Code answered 93.4% of baseline tasks correctly; Codex answered 62.1%. When the researchers added the original papers, overall accuracy improved slightly. Claude Code's accuracy on tasks that could not be reproduced fell from 100% to 63.3%.
Paper access helped with ordinary execution. In the missing-material cases, it also supplied numbers that the package could not support.
What the benchmark holds still
The collection covers 13 subject areas, five online repositories, and work written in R, Python, and Stata. Each paper contributes no more than three tasks. An agent has to understand the original analysis, make the environment run, execute the pipeline, and extract a requested result.
The score is not an estimate of how much social science reproduces. Before building the tasks, the researchers manually ran the included analyses three times and retained results that were identical each time. They also included ten cases with missing code or restricted data. Those cases have a deliberate correct answer: reproduction is not possible from this package.
The repository publishes tasks, gold answers, research-question annotations, metadata, and analysis scripts under CC BY 4.0. The team's Harvard Dataverse hosts the anonymized replication packages. Removing titles and author details makes direct recall less useful.
The tested systems were Claude Opus 4.6 through Claude Code and GPT-5.3-Codex through the Codex CLI. Each ran three times in a sandbox, without web search, access beyond the supplied directory, memory of earlier runs, or human intervention. Package installation was allowed.
The prompts differed slightly. Codex received extra instructions to construct an executable replication script because it did not reliably repair the packages by itself. The comparison therefore covers two complete agent setups. It does not isolate the underlying models.
Old software drove much of the gap
Claude Code reached 93.4% accuracy across tasks and fully reproduced 78.0% of papers. A paper counted only when all its tasks were correct. Codex reached 62.1% and 35.8%, respectively.
Original research archives are often awkward to rerun. In this sample, missing dependencies sat beside hardcoded paths, incomplete environment descriptions, deprecated APIs, and notebook constraints. Claude Code worked through those problems in all three runs. Codex returned no answer for 17.8% of tasks and failed on 27.0% of papers.
Outright failure was only part of the difference. Among tasks Codex completed, accuracy was 75.5%; among completed papers, 49.2%. About one in four completed tasks still had a wrong answer.
The programming-language split should be read descriptively because the groups are unequal: 136 R tasks, 49 Python tasks, and 36 Stata tasks. Claude Code scored 91.9% on R, 100% on Python, and 94.4% on Stata. Codex ranged from 40.0% on Python to 69.1% on R. Environment repair and the particular packages in the sample are mixed into those figures.
More context weakened the refusal
Giving each agent the original PDF improved normal execution. Claude Code rose from 93.4% to 94.5% task accuracy, and Codex rose from 62.1% to 65.4%. Codex's paper failure rate dropped from 27.0% to 5.6%. Information in the paper may have clarified file order, dependencies, or expected output.
The ten missing-material cases moved the other way. Claude Code fell from 100% to 63.3%. Codex fell from 100% to 90.0%. Instead of reporting that the supplied package could not support the result, an agent sometimes returned the number printed in the paper.
The researchers then used a more pointed prompt. They asked the agents to explore "alternative analytically defensible approaches" and choose results that most closely matched the publication. Overall task accuracy rose to 94.1% for Claude Code and 74.1% for Codex. Codex's task failure rate dropped from 17.8% to 0.5%, and its paper failure rate dropped from 27.0% to 1.9%.
On the tasks that could not be reproduced, Claude Code managed 70.0% and Codex 60.0%. According to the paper, both systems sometimes filled absent evidence with plausible numbers taken from the PDF. Pressure to persist fixed some executable problems while making a scientifically correct stop less likely.
Show the answer only after the blind run
A reliable workflow separates execution from comparison. The first run gets the code, data, and task, plus explicit permission to return NOT_REPRODUCIBLE. It does not get the published coefficient, table, or conclusion. Its record should contain the commands it ran, changes to the environment, output files, extracted values, and the reason for stopping when it cannot continue. A matching number without that trail proves little.
Once those artifacts are locked, a second reviewer can see the paper, compare outputs, and classify the discrepancy without steering the original execution. The scoring must accept a justified refusal. There were only ten missing-material tasks, yet those ten cases tested whether the system preserved the line between unavailable evidence and an answer.
The same separation helps outside research. In BLOGish's test of prompt-injected payment agents, apparent task completion hid a security failure. READY prices human review as part of agent reliability, which matters when an agent rewrites scripts or chooses between plausible specifications. A team using api.ish.chat can keep this blind workflow fixed while comparing whether models stop on the same missing-evidence cases.
The limits are part of the result
SocSci-Repro-Bench is a preprint with 54 papers, two agent configurations, and only ten non-reproducible tasks. Its language groups are uneven, its prompts differ, and its main tasks were selected because people had already made them reproduce. The percentages do not describe all social science, a random deposit of replication packages, or later versions of either agent.
The benchmark separates two abilities that one success score can blur. An agent may repair old code exceptionally well and still make a poor independent auditor once it knows the expected answer. The safe handoff point is concrete: show it the paper only after it has committed to what the package can prove.
