TechforHumans approved 3 of 8 models. Its replay gate is the result worth copying
A model swap should be tested against the work your agent actually does. One small production-parity study shows how, and where its own verdict can mislead.
TechforHumans put eight models through hundreds of replays of a conversational-agent workflow. Three passed: GPT-5.4 mini, GPT-5.4 nano, and Kimi-K2.5. The ranking is not the most useful result. Three other candidates scored above 79% on the study's composite score and were still rejected because their hallucination rate crossed a separate safety threshold.
That decision structure deserves attention. A weighted average lets a cheap, fast model numerically compensate for a dangerous answer. A gate prevents speed from offsetting a failure the team has defined as unacceptable. The company's full account of the evaluation also supplies the caveats: only 20 curated sessions formed the reference set, tool execution was simulated, and the reported winners belong to this workflow.
Replay the product, not a cleaned-up demo
The pipeline started with 106 synthetic customer-service sessions derived from real interaction patterns. A human reviewer checked routing, instruction adherence, message order, and tool calls. Only 20 sessions across four service instances survived as the approved reference, called Base 0.
For each trial, the replay engine reconstructed the original memory, prompt, conversation history, available actions, skill changes, and termination rule. It replaced only the model making the next decision. This is what the authors mean by production parity. The candidate did not receive the expected answer in its prompt.
Copying a user message into a playground would miss much of this. Agent behavior depends on the harness around the model. An omitted memory field, a different tool schema, or a history filter can change the action. Anthropic's guide to agent evaluations draws the same boundary: evaluating an agent means evaluating the model and its harness together, while a complete transcript includes intermediate tool calls as well as the final response.
To make a fair swap test, freeze everything except the candidate model. Version the system prompt, tool definitions, memory transformation, provider adapter, retry policy, and stop condition with every run. If two candidates see different operational contexts, the experiment compares two systems, not two models.
Do not let a replay touch the real world
TechforHumans did not let a candidate cancel a policy or alter a customer record. When the model chose the expected tool at the expected point, the harness returned the historical tool response. When it diverged, the harness returned a simulated failure and discouraged another attempt. Tool calls were capped at five per turn.
That boundary keeps a replay safe, but it introduces bias. A different sequence may be valid in production and still receive a generic failure because Base 0 did not take that path. The study partly addressed this with a contextual judge that could credit a sensible alternative. The judge still cannot reproduce the state transition of an unexecuted tool.
A replay suite needs to treat actions according to their effects:
- Read-only calls can run against a fixed snapshot or deterministic stub.
- Mutating calls should write to an isolated sandbox whose final state can be inspected.
- Irreversible or external calls should be intercepted and graded from their proposed arguments.
The end state matters more than a polished closing sentence. The tau-bench paper evaluates tool agents by comparing the database state after a conversation with an annotated goal state. That catches an agent that claims a booking succeeded when no reservation exists. For syntax and tool selection, the Berkeley Function-Calling Leaderboard is a useful public reference. An internal replay should add the policies and state transitions particular to the product.
Put vetoes outside the average
TechforHumans combined quality, normalized latency, and normalized cost into a Final Score, weighted 50%, 20%, and 30%. Approval also required at least 75% Final Score and 70% Pass Rate. Independent hallucination gates came next: zero critical hallucinations and no more than 2% overall.
Those gates changed the verdict. Gemini 2.5 Flash scored 79.7%, GPT-4.1 nano 82.9%, and GPT-5 mini 79.2%, yet all three exceeded the overall hallucination limit. GPT-5 mini also recorded the only critical hallucination rate in the table, 1.6%. A composite ranking alone would have hidden why they were rejected.
Each product needs its own veto list. For a coding agent, it might include modifying files outside the repository, exposing a secret, or reporting tests as passed when they did not run. For a support agent, it might include inventing a refund, bypassing identity checks, or failing to escalate a regulated request. A violation should fail the trial directly, with the ordinary quality score retained for diagnosis.
Human calibration belongs here too. The study used multiple model judges, took the median for quality, and used the most severe judge classification for hallucination. That is conservative, but a panel can share blind spots. Review a stratified sample of passes, failures, and judge disagreements, then measure the judges against those human decisions. This keeps subjective grading visible and testable.
Repeat the verdict, not just the prompt
The aggregate table looked decisive. Individual runs were less tidy. Gemini 2.5 Flash and GPT-5 mini moved between approval and rejection across repeated runs, while GPT-OSS-120B never passed. The latter also had a concrete integration failure: the harness failed to recognize a tool call or text decision in 87% of sessions because of its Harmony-format output.
The first pattern points to sampling uncertainty near a threshold. The second points to a broken adapter. Mixing provider errors, parsing failures, policy errors, and reasoning failures into one rate makes the next fix harder to identify.
Store a failure taxonomy beside every trace. Report repeated-trial reliability, confidence intervals where the sample permits them, and the verdict distribution across runs. One mean cannot answer whether a model is reliably safe enough to ship.
A small suite can still earn its keep
Twenty sessions are too few to establish a general winner. They may still be valuable regression tests for the four flows they cover. Publish a coverage map: which intents, tools, languages, user behaviors, and failure severities appear, and which do not. Add production incidents and newly discovered edge cases without silently rewriting the historical set.
Cost telemetry should be literal too. TechforHumans notes that its normalized cost score used token volume, not dollars. Recomputing with public list prices changed the ordering, with the reported estimated cost per session for GPT-5.4 mini about 5.3 times GPT-5.4 nano. If a proxy can reverse a purchasing conclusion, readers need to know exactly what it measures.
A model-swap evaluation should end with a reversible deployment decision. First replay the frozen suite, reject any hard-gate violation, and inspect the traces. Then canary the surviving model, monitor production drift, and keep the previous configuration ready to restore. The same discipline helps when comparing models behind api.ish.chat, where a compatible endpoint does not make behavior interchangeable.
Public benchmarks measure models under their own rules. A production replay checks what your complete system did under yours. TechforHumans' small study is useful as a worked example of that difference, not as permission to copy its winner.
