GameXpert-Bench: the best strict repair score was 39.0
A coding agent can produce a game that starts, responds to input, and looks convincing in a screenshot. Maintenance is less forgiving. The agent has to find defects nobody listed, repair them without breaking nearby behavior, and keep later product changes coherent.
GameXpert-Bench, a new preprint from researchers affiliated with Tencent Hunyuan, Tencent Lightspeed Studios, and several universities, measures those later jobs alongside initial generation. Its most useful result is the gap between following a bug checklist and discovering problems independently.
The strongest model on the GameFix track scored 39.0 on STRICT, a metric centered on tasks where an agent repairs nearly all injected bugs. The median was near 14. A polished first draft tells us little about whether the same agent can maintain what it created.
Three jobs that should not share one score
GameXpert-Bench has three tracks:
- GameGen starts in an empty workspace and asks for a complete game from one natural-language request. It has 97 tasks across 11 genres, including 53 2D and 44 3D games.
- GameFix uses 50 human-verified internal game levels. The benchmark injects 19 to 27 reversible bugs into each level, then runs one task with the full issue list and another with most objective defects hidden. That produces 100 repair tasks per run.
- GameOpt replays six cumulative improvement requests across 17 JavaScript games. Its 102 turns are checked against 701 criteria covering gameplay, level design, balance, art, interface, audio, and regressions.
Each track grades a different kind of work. GameGen combines automated behavior checks, code inspection, live interaction, and human assessment. GameFix runs deterministic Fail-to-Pass and Pass-to-Pass tests in a network-disabled browser sandbox. GameOpt evaluates the final product with evidence-oriented rubrics. Comparing their raw scores would be meaningless.
Many leaderboards compress coding into one figure. These three tracks instead ask whether an agent can create, maintain, and improve the same kind of executable product.
What disappears with the checklist
GameFix uses two prompts over the same broken games. Explicit Issue lists every unreasonable behavior. Self-Discovery withholds objective defects such as reversed controls, impossible movement speeds, and blocked paths. The agent knows that other strange behavior exists, but it has to find the problems.
With the checklist, the 17 evaluated models sit inside a range of about 13 points. Under self-discovery, that range expands to about 38 points. Claude Opus 5 leads STRICT at 39.0, followed by Claude Fable 5 at 33.7 and GPT-5.6-sol in Codex at 29.1. The researchers ran each setup three times and used average@3 survival curves instead of reporting one favorable pass.
Without the checklist, code editing turns into a larger investigation. The agent must explore runtime behavior, judge whether something is broken, infer intended values, repair multiple sites, check for regressions, and decide when to stop. Some agents found additional defects but refused to change them because they were not explicitly authorized. Others repaired the visible symptoms and ended the task.
Teams should treat "fix these five bugs" and "find what is wrong" as different assignments. The second needs an observable completion signal, perhaps executable tests, a coverage target, a review protocol, or a human decision. Otherwise, the agent can report real progress while leaving the most difficult defects untouched.
The first playable version is still a draft
The leading GameGen model scored 79.7 overall and 94.4 for completeness, but 72.0 for richness and 72.4 for player experience. Richness was lower than completeness for all 15 models. Agents built the required gameplay skeleton more reliably than optional mechanics, deeper content, and polished interaction.
Three-dimensional work cost more. The cross-model average fell from 65.9 on 2D games to 60.1 on 3D games, and 14 of 15 models performed worse on the 3D subset. Completeness had the largest average drop, not visual quality.
Both tracks expose the same failure: plausible components do not prove that a product behaves correctly during use. The same applies to dashboards, workflows, and APIs. Evaluation should follow the behavior users encounter, not stop at a successful build or static diff. BLOGish's guide to coding-agent trajectory evaluations develops that approach for general software work.
High optimization scores need context
GameOpt gives its leaders high scores, including 93.96 for Claude Opus 5. That is a weighted final-product result over 17 fixed game chains. It does not mean the model solved 93.96% of arbitrary game-development tasks.
Scores also varied by product dimension. Across 15 models, balance and level design had the lowest means, while audio had the highest. Results over six turns were non-monotonic, and request type was confounded with turn order. Audio always came at turn six, so the late rebound cannot establish better long-context behavior.
Regression checks are the interesting part of the design. Every request is judged against new acceptance criteria and behavior that should survive from earlier versions. Repeated runs matter for the same reason. A single working artifact cannot establish consistency, as the CAR-bench pass3 analysis explains.
Reproduction has to wait for the artifacts
The paper includes substantial methodological detail, tables, prompts, and examples. The public project page currently marks the paper, code, and dataset links as "coming soon." The GameFix Gold Games are confidential and will remain internal to reduce contamination.
The reported findings are still worth examining, but outside teams cannot audit the complete benchmark today. Treat its leaderboard as paper-reported until the promised artifacts arrive. GameDevBench, by comparison, publishes its evaluation code and camera-ready results. GameCraft-Bench has released its Godot harness and data. Their scores are not directly comparable because their tasks and grading differ, but their repositories let researchers inspect and rerun the machinery.
For an internal evaluation, copy the separation rather than the leaderboard: test creation, hidden-defect discovery, regression-safe repair, and cumulative improvement as distinct jobs. Then remove the checklist and see whether the agent can prove it is finished.
