SWE Refactor Bench found green tests can hide a missing migration
A migration has an awkward definition of success. The internals must change while the observable product does not. That makes the usual red-to-green coding benchmark a poor fit. The original repository already passes its tests. If an agent returns the code untouched, every behavioral check may stay green even though the requested work never happened.
The authors of SWE Refactor Bench call this evaluation failure “Blindness.” Their benchmark, submitted on August 24, 2026, asks coding agents to complete 20 whole-repository migrations covering language rewrites, framework rewrites, platform ports, and build-toolchain changes. Across 520 runs from eight frontier models and 26 model-effort configurations, only 28 runs, or 5.4%, passed all three evaluation stages. Thirteen of the 20 tasks received no accepted solution.
The grading design is the useful part. It separates three questions that engineering teams often bundle together: Did the migration happen? Did the known behavior survive? Can a fresh adversary find a difference the fixed suite missed?
A green suite answers only one question
Bug-fixing benchmarks start with a known failure. A successful patch changes a specific test from red to green, so doing nothing earns nothing. Behavior-preserving migration is different. The desired before and after should produce the same outputs, artifacts, API responses, and installation layout.
SWE Refactor Bench makes the problem concrete. One task asks for cmark 0.31.1 to move from C to Rust 1.90 while preserving its C ABI and release artifacts. In a run examined by the paper, all 4,184 fixed checks passed. The migration audit still rejected it. The Rust version reproduced the C implementation's layout and unsafe pointer machinery closely enough to count as a transliteration rather than the required replacement.
Other failures are simpler. A submission can preserve the old dependency, keep legacy code on the default path, or hide a prebuilt reference implementation behind a new wrapper. A behavioral suite may confirm that the product still works while having no way to establish which implementation produced the result.
This does not require a scheming agent. The score is missing a measurement. More tests of the same kind cannot prove that the old stack left the tree.
The benchmark uses three vetoes
The open-source benchmark repository publishes all 20 task definitions and its Apache-2.0 grading harness. Its task schema documents the same three gates used in the released corpus:
- Migration Audit: A model examines the original and submitted source without executing either. It checks five to ten required criteria per task, including whether the target implementation is on the default path and whether the old toolchain, dependencies, and release path are gone. Three samples vote on each criterion. One failed criterion gives the task a zero.
- Behavioral Tests: Fixed, rule-based checks build both versions and compare observable behavior. A submission must pass every check to receive 40 points and proceed.
- Agentic Verification: Six independent coding agents each get one hour with both built trees. They must construct an input that passes on the original and fails on the migration. A counterexample counts only when it reproduces three times. Each verifier that finds no break contributes ten points.
The corpus contains 130,118 fixed behavioral checks. That large suite was not sufficient by itself. Of 520 runs, 340 passed the migration audit and 118 passed every fixed check, but only 88 did both. The six verifiers reduced those 88 to 28 accepted runs.
Each gate catches a different failure. Some submissions preserved behavior by skipping part of the migration. Most attempted the migration and broke known behavior. Others cleared the audit and fixed tests, then failed on a new executable counterexample.
The model table is not the main lesson
The official leaderboard reports that the strongest configuration in this study, Claude Opus 5 at xhigh effort, scored 47.0 out of 100 and averaged $74.90 in API spend per task. GPT-5.6 Sol at max effort scored 28.5 at an average $143.50 per task. These are measurements from this harness, model version, task mix, and run policy, not universal price-performance rankings.
Build-toolchain rewrites earned a mean score of 31.4, while language rewrites averaged 5.6. Framework rewrites produced 14 of the 28 accepted runs. “Repository migration” should not be treated as one capability when moving Maven to Gradle and moving C to Java pose different problems.
Results also depend on the verifier panel. The paper reports that removing its two strongest verifiers would have accepted 46 submissions instead of 28. An accepted submission is not formally proven correct. It survived the strongest adversaries the researchers used.
The study has other boundaries. The benchmark contains 20 frozen open-source projects and one run for each task-configuration pair. Its migration audit uses a model judge. A human audit of 2,040 criterion decisions found 133 disagreements, a 6.5% error rate, with most errors making the judge too strict. Those choices support a reproducible, conservative study, but one score still should not decide a purchase.
Our review of RuBench's served-model identity failure reaches a related conclusion: before comparing model labels, establish what the harness actually measured. WebDev-Skills-Bench found that extra agent instructions raised token use without improving the reported task outcome. Harness design can dominate the story told by a leaderboard.
Build a migration acceptance plan with three kinds of evidence
Engineering teams do not need 130,118 checks to borrow the useful idea. They do need separate gates.
First, define structural exit criteria. Name the packages, runtimes, build tools, generated artifacts, compatibility shims, and production paths that must disappear. Verify the default deployment path, not merely the presence of new code. Source scans, dependency lockfile checks, artifact inspection, and clean-room builds can make these requirements executable.
Second, preserve known behavior. Record outputs from the original system before work begins. Include API contracts, command-line behavior, serialization quirks, error text where callers depend on it, installation paths, and performance or resource limits that matter in production. Hide some acceptance cases from the implementing agent.
Third, fund adversarial comparison after the fixed suite passes. Use differential fuzzing, shadow traffic, property-based tests, or a separate agent asked only to find disagreements. Give the verifier both systems and require a reproducible input, not a prose objection. When testing several models through a compatible endpoint such as api.ish.chat, pin the model identifier, harness version, effort, budget, and tool permissions so the comparison can be rerun.
The owner of the migration should not be the only owner of its proof. Let one process implement, another check structural completion, and a third search for behavioral differences.
Migration is a two-axis job
“The tests pass” says the new tree preserved what the suite knows how to observe. It does not say the old stack is gone, the new stack is actually serving production, or an unanticipated input will behave the same way.
SWE Refactor Bench shows that migration completeness and behavioral preservation are different abilities. A credible modernization plan must measure both, then keep looking for counterexamples after they appear to agree.
