FuXi reports a 19-task tie. Its public runner covers 11
Two coding agents finish every task in a benchmark. The score says they tied. It does not say whether their abilities are equal or whether the tasks ran out before the agents did. FuXi's new comparison cannot separate those explanations.
The FuXi benchmark report gives FuXi with a model labeled deepseek and Claude Code with claude-opus-5 the same result: 19 tasks passed, zero failed. Fifteen are small Python exercises. Four use a described 25-plus-file order-management project. The report calls the systems statistically indistinguishable on the suite and claims a cost advantage for FuXi's model.
The repository publishes scenarios, tests, shell runners, and result summaries rather than only a chart. Read together, those artifacts support a narrower finding than the report's headline.
The public artifacts stop at four different lines
The report describes 15 micro tasks plus four large-project tasks. The repository contains directories for all 15 micro scenarios, covering bug repair, feature work, refactoring, test generation, review, data structures, validation, and documentation. It does not contain the reported orderapp project or artifacts for the four large-project runs.
The two published result files, one for FuXi with deepseek and one for Claude Code with claude-opus-5, cover the 15 micro scenarios. Each entry records a baseline summary, a final test summary, and the client's exit status. Both files end with an extra closing brace, so a strict JSON parser rejects them. The formatting bug is easy to fix. It still gets in the way of the files' stated job as machine-readable raw results.
The public FuXi runner and matching Claude runner go one step shorter. Their prompt maps and loops cover d1 through d11. The result summaries also include d12 through d15, but the repository does not show the commands that produced those four entries. It publishes no per-run logs.
The executable creates another mismatch. The methodology names FuXi 0.15 and labels it a planned release. The latest public binary release is 0.1.2, published August 6, 2026. FuXi's license says its source and internal implementation are proprietary, even though the benchmark fixtures and documentation are public. A reader can inspect the tests, but cannot obtain the benchmarked client version from the linked release or build it from this repository.
These mismatches prevent an outside evaluator from reproducing all 19 reported tasks with the supplied files. They do not show that the runs are false.
Nineteen green checks do not establish parity
Both systems passed everything they were scored on, leaving no failed or discordant case that could separate them. The benchmark discloses that each scenario ran once. It reports neither repeated samples nor confidence intervals.
Several tasks also expose their tests to the agent and explicitly ask it to run them until they pass. This is a legitimate way to measure a repair loop. It does not measure whether a change generalizes beyond visible assertions. Our review of SWE Refactor Bench showed why that distinction matters: a patch can preserve every supplied test while missing part of the intended migration.
The refactoring task makes the scoring limit especially clear. Its baseline already has five passing tests. The prompt asks the agent to reduce duplication and improve naming while keeping behavior unchanged. Five green tests after the edit show preserved behavior. They do not score whether the code became clearer or less repetitive.
Test generation leaves another open question. The FuXi result reports 71 passing tests and 100% coverage. The Claude result has 102 tests and the same coverage. Neither number establishes which suite catches more faults. Statement coverage cannot show whether the tests assert the right behavior, and a larger count may come from splitting cases.
The report also says the cheaper model produces a substantial cost advantage, but it publishes no token counts, per-run prices, total spend, or latency. Provider labels alone cannot fill that gap. The report acknowledges that both model identities come from configuration or a proxy rather than independent verification. RuBench's fallback-model failure is the practical warning: the requested model name is not evidence of what an endpoint actually served.
What the benchmark does show
The public results have a practical use. According to the supplied summaries, both client-model combinations completed 15 compact Python scenarios and left the visible tests green. The scenarios are small enough to inspect, and most expected behavior is explicit. A team can adapt them into smoke tests for its own endpoint.
FuXi's README calls the comparison small and self-run, says it is not a third-party benchmark, and tells readers to treat it as a data point. A vendor test can be informative. This one does not contain enough separation or cost evidence to support the larger claims about comparable capability and lower cost.
The same discipline applies to model-format comparisons. A surprising result can be real while the label hides another changed variable, as the 4-bit versus BF16 comparison demonstrated. A useful report identifies every moving part before it ranks the output.
A benchmark should be runnable from the claim down
Every reported result should trace back to runnable public artifacts.
- Start with released client versions and immutable scenario commits. An unreleased tested binary needs a published artifact and checksums before the comparison can be rerun.
- The run record should include the requested model, endpoint, response metadata, and provider-specific routing. A self-hosted or compatible API must expose enough identity evidence to audit what it served.
- Use one published runner for every reported task. Include exact prompts, clean baselines, environment setup, valid raw output, and complete logs.
- Visible tests need a second measure. Refactoring can use a structural rubric, while generated tests can be checked with mutation or seeded faults.
- Repeat each system. Report per-task outcomes, variance, token use, wall time, and actual spend instead of inferring cost from a model name.
FuXi's public micro scenarios are a usable start for such a harness. The published summaries say two agent setups cleared a small visible-test suite. Extending that result to capability parity requires the missing runners, large-project artifacts, released client, verified model identity, repeated runs, and cost records.
