FM-Bench ran agents for 20 football seasons. None learned the transfer market's hidden price
FM-Bench hands an AI agent a football club, 26 tools, and roughly 340 to 400 decision stops across 20 simulated years. The agent signs players, negotiates contracts, invests in facilities and youth, picks lineups, manages cash, and tries not to get fired. The matches are fictional. The management problem is familiar: information is incomplete, competitors react, and a safe decision today can weaken the club five seasons later.
In the FM-Bench paper, claude-fable-5 leads both the solo track and the shared Arena. The field's common failure is more instructive. After hundreds of rejected bids, none of the 15 tested models learned the market's hidden acceptance prices.
The failure took time to become visible. Many agent benchmarks would have ended before it did.
Year five picked the wrong leader
Each solo run places one model against 15 scripted clubs in a deterministic world. The researchers ran three seeds for each of 15 models. Every model run reached the end of 20 years, while the blind scripted anchors died out in seven of nine runs. A deterministic engine calculates the cumulative score without an LLM judge or human grader.
On seed 1, the correlation between the year-five ranking and the final order was only 0.19. It reached 0.78 by year 15. deepseek-v4-pro led after five and ten years, then finished 12th. Stopping after five seasons would have rewarded a different management style.
Delay is built into ordinary choices. Youth and facility spending can take years to pay back. A large wage bill may eventually provoke the board. Cash feels safe but contributes less than a stronger squad. Rivals raise transfer demands after repeated bids, so a fixed tactic gets worse as the agent keeps using it.
The official launch article describes the two tracks. Solo holds the scripted world fixed for each tested model. Arena puts 15 models and one scripted anchor into a shared 20-year economy where signing a player removes that player from everyone else's options. Ten models won the league at least once in the Arena. The eventual composite winner took four titles.
Thirty offers, no durable price estimate
A seller's hidden threshold rises after each rejection. The privileged oracle knows that threshold and closes a signing in one offer. The model field needed a median of 30 offers per completed signing. claude-fable-5 did best at nine. gemini-3.5-flash needed 73, and one seed reached 133.
Repeated failure did not produce a stable estimate. The agents saw rejection after rejection, but later bids did not converge on the acceptance boundary. They did not lack a fact that could be retrieved. They failed to update a policy during the run.
There is a recognizable production failure here. An agent receives feedback, preserves a trace, and starts the next task under the rule that caused the last mistake. A larger context window may retain more rejected bids without changing the next bid.
Each FM-Bench decision stop begins a fresh conversation. The engine supplies current state and recent events, while a self-authored notebook carries the agent's plans across stops. Notebook snapshots show two bad extremes. gpt-5.6-sol had 0.91 TF-IDF cosine similarity between seasons, consistent with an archive that mostly grows. claude-sonnet-5 scored 0.20 and qwen3.7-max 0.23, consistent with plans being replaced so thoroughly that little survives. The winner sat at 0.39, against a field median of 0.31. The paper cautions that similarity alone does not establish good memory.
BLOGish previously examined funes and agent-owned transcript memory. FM-Bench shows the next problem. Storing the history is easier than deciding which evidence should change a durable rule.
Seven times the tokens, no better order
Token use ranged from 28 million for gpt-5.6-terra to 194 million for deepseek-v4-pro, a factor of seven. The reported overall Spearman correlation between tokens and score was -0.19, and no seed showed a useful positive relationship. The winner was among the three cheapest models.
The result does not establish that extra reasoning is useless. This harness did not convert more tokens into better long-range control. Boundary-Bench found that security hardening can sharply raise agent cost. FM-Bench supplies another cost warning: high spend does not show that an agent revised its policy or used feedback well.
Higher-scoring models reduced slow-payoff investment near the end, kept less cash idle, and started contract renewals earlier. Each measure tracked score across all three seeds. Those behaviors say more than the provider name attached to a row.
The reproducible part is solo
The open repository includes the deterministic engine, solo runner, scripted baselines, scoring code, rules, a public reference world, and replay and truth-isolation tests. Result files carry engine_commit, params_hash, and the score version. A five-year heuristic baseline runs locally:
python -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python run_benchmark.py --agent heuristic --seeds 1 2 3 --years 5
The repository is a v0 research preview. Analogy AI operates the official Arena, and the official hidden seeds and trajectories are not public. Human players currently have fewer actions because append_note, set_standing_order, and the optional draft phase are not fully available in the web interface. The Arena result comes from one seed with no error bars, so positions separated by a few points should be treated as ties.
Independent teams can do more with the solo track. Connect several models, including models reached through api.ish.chat, and compare replayable action logs under the same engine. Football can also be replaced with a domain where decisions compound, such as incident response, inventory, customer retention, or infrastructure maintenance.
Keep the long horizon and deterministic replay, then measure whether feedback changes policy. An agent can remember every rejection and still make the same bid.
