RuBench caught a coding benchmark measuring the fallback model
A benchmark row may name an agent, a model, a reasoning level, and a score. RuBench found a case where the named model was not the one that completed the task.
The benchmark contains 25 repository-maintenance tasks written natively in Russian. During an out-of-competition Claude Code and Fable 5 run, five tasks triggered an official safety fallback to Opus 4.8. The requested model and serving model had diverged, so the authors excluded those cells from the Fable result.
This is more informative than another small leaderboard. It shows that agent evaluations need receipts for the system that ran: serving model, harness, tools, policy events, and network activity. Without them, a model comparison can quietly become a comparison of routing decisions.
Russian requirements written as Russian requirements
The RuBench paper addresses a gap that translation cannot fully close. Repository benchmarks usually derive specifications from English GitHub issues. RuBench's task statements were written from scratch in Russian, in the style of customer requests, rather than translated from English.
Its 25 tasks come from fixes in aiohttp, aiogram, Laravel, NestJS, and Fastify. They span Python, PHP, TypeScript, and JavaScript. The project says every fix commit came after the training-data cutoff of each evaluated model. Upstream maintainers' regression tests grade the patches. The tests and gold fixes remain private, but the project published a SHA-256 commitment to the grading oracles before release.
Readers can inspect the public dataset, which contains the Russian specifications and metadata. The repository also publishes complete trajectories, final diffs, per-cell metadata, and results. The fallback finding came from those traces, not from the final percentages.
Native writing matters because translation can tidy ambiguity, normalize tone, or replace the vocabulary developers actually use with benchmark English. RuBench tests intent recovery in one real language setting. It does not prove multilingual competence, but it is more faithful to that setting than a translated issue.
A ranking with wide uncertainty
The first round ran three repetitions for four ranked product configurations. Claude Code with Opus 4.8 scored 78.7%, Claude Code with Sonnet 5 scored 74.7%, Codex CLI with GPT-5.5 scored 66.7%, and Claude Code with Haiku 4.5 scored 53.3%.
With only 25 tasks, most adjacent gaps were not statistically resolvable. The authors say only differences against the weakest configuration cleared that bar. The exact ordering is less informative than the setup: fresh fixes, repeated runs, executable tests, pinned CLI versions, and public trajectories.
Readers naturally treat each line in a sorted table as a distinct rank. Here, the defensible conclusion is narrower. These deployed configurations behaved differently on this small Russian task set, but the top rows were not clearly separated.
Five HTTP fixes changed models
The out-of-competition Claude Code and Fable 5 run exposed the stranger result. Five of 25 tasks, described in the paper as routine HTTP-protocol fixes, activated the product's safeguard path and were completed by Opus 4.8. Only 20 cells remained measurable as Fable 5, with 17 passes. Folding the substituted cells into one Fable score would have credited it for another model's work.
Anthropic's fallback guide documents the behavior. Fable 5 uses conservative safety classifiers in areas including cybersecurity, so benign technical work can trigger a fallback. The response identifies the model that answered, while usage.iterations records each attempt. Anthropic advises analytics systems to use the served model rather than merely the requested one.
The substitutions were documented product behavior, not a secret mechanism. A simple benchmark label could still conceal them. Response metadata and trajectory inspection were necessary to make the row's provenance accurate.
Evidence that should travel with the score
Start with the requested model and the model that served every turn. Store both for retries and subagents too. Then pin the harness version, because a CLI update can alter tools, prompts, context handling, or routing while the model name remains unchanged.
Reasoning level belongs in the record. So do refusals, classifier categories, and fallback markers. If a policy layer substitutes a model, report those cells separately rather than blending them into the requested model's score.
Tool activity also changes what a pass means. A run that retrieves an upstream fix is no longer solving the same closed task as one confined to the supplied repository. Network calls and file reads should remain available for audit. Finally, publish task-level outcomes so readers can calculate paired comparisons and confidence intervals instead of relying on one aggregate percentage.
The rule still applies when evaluation traffic goes through api.ish.chat or another gateway. Keep the requested route and the serving metadata returned downstream. A unified interface should simplify calls without erasing provenance.
Our analysis of AgentX as an inference cache and scheduling test made a related point: the measured unit includes routing, cache state, pauses, and parallel work. A coding benchmark adds the harness, its tool policy, and its safety fallbacks.
Useful evidence, narrow scope
RuBench is a one-author benchmark covering 25 tasks, one natural language, and five repositories. Its specifications were reconstructed from real fixes, not collected from Russian users in a live study. The grading tests are withheld, so readers can audit the hash commitment and released artifacts but cannot reproduce the full judge today. Its scores do not represent all Russian software work or every agent configuration.
The native specifications make one evaluation setting more realistic. The full trajectories then reveal when a model label has stopped being accurate. Both choices are worth copying.
For future agent comparisons, the results table needs a field for the serving model and a count of substituted turns or cells. If those differ from the requested model, the benchmark should say so before anyone cites the score.
