EvalAgent reached 65% Eval@1. One third of its generated evaluators still needed debugging
A coding agent can make an evaluation project look serious before any evaluation happens. It writes a long plan, invents a dozen metrics, adds test files and a report generator, then produces an entry point that crashes. Sometimes it runs and every score is zero.
AWS AI Labs studied this problem by asking coding assistants to build agent evaluations from source code, execution traces, and a short requirement. Its proposed system, EvalAgent, reached 65.0% Eval@1 with Claude Sonnet 4.5. Eval@1 requires the generated evaluation to execute and return substantive results on the first attempt.
That was the best result in the experiment. It also means 35% still needed manual debugging. An evaluator written by an agent is another program, and it needs to earn trust before its scores do.
The evaluator can fail independently of the agent
An integration test usually begins with known behavior and a known assertion. Agent evaluation has an earlier design problem: someone must choose the behavior worth measuring. The resulting code then has to extract evidence from a variable trajectory and turn it into a meaningful score.
The researchers assembled AgentEvalBench from 20 agents, nine frameworks, and 14 domains. The set included a browser agent, medical document processor, code assistant, travel planner, and network switch operator. Every agent had pre-collected traces and two kinds of requirement. The generic version asked for evaluation code without naming metrics. The specific version named a concern such as medical entity correctness or recommendation precision and recall.
Four baselines separated the effects of source access, traces, and planning. A single-turn model received everything in one prompt. The agentic variants used Claude Code with source alone, source plus traces, or a plan followed by implementation. EvalAgent added procedural skills, reusable templates and trace-processing code, plus current API documentation retrieved through Context7.
The agentic approaches shared the same Claude Code version and a limit of 50 tool calls. Claude Haiku 4.5 and Sonnet 4.5 supplied the evaluator backbones. The results compare scaffolds inside one model family. They do not show that the ordering will survive a different coding model.
A 584-line plan did not keep the code focused
Under generic requirements with Sonnet, the unstructured two-stage baseline wrote an average 584-line plan and 1,902 lines of evaluation code. EvalAgent averaged 134 plan lines and 289 code lines.
Most of the extra material was not useful. Across 20 agents, the two-stage baseline produced 29 dead-code artifacts, including 11 report generators that its entry point never used. Plans named one method while implementations substituted simpler keyword checks.
Planning had become an invitation to add scope. EvalAgent's skills required a small set of distinct metrics, a minimal working implementation, shared trace parsers, and code that followed the written plan. Nineteen of its 20 projects settled into a consistent two-file pattern.
Ecdysis' grouped harness repair reached a related conclusion about procedural knowledge. It helped by narrowing decisions and reusing tested machinery. Adding more instruction text would not provide the same benefit by itself.
Better evidence made the implementation harder
Source code shows which tools an agent may call. A trace shows what happened: the chosen tool, its arguments, the response, and the next decision. Those details are necessary when a plausible final answer may have followed a broken process.
The direct trace-based baseline earned 73% to 77% win-tie rates over source-only evaluation in the paper's quality comparisons. Yet Sonnet's Eval@1 dropped from 60.0% with source-only evaluation to 35.0% with traces. Heterogeneous trace data created more opportunities for parsing mistakes.
This problem also appears in NIKA's trajectory submissions. Trajectories reveal whether an agent diagnosed an incident or guessed, but the team still needs reliable code to interpret them.
Reusable extraction patterns helped EvalAgent carry that complexity. When the prompt requested five metrics, EvalAgent reached 65.0% Eval@1. The direct and planned baselines reached 30.0% and 40.0%. Without a shared parser, every new metric could add another incompatible view of the same trace.
Fresh documentation prevented stale integrations
EvalAgent queried current library documentation while generating code. Removing Context7 cut Sonnet's Eval@1 from 65.0% to 20.0% in the paper's ablation. Generated evaluations otherwise used wrong model identifiers and outdated integration patterns for DeepEval, LiteLLM, and Bedrock.
Documentation did not prevent every integration error. Nine of Sonnet's 14 failures implemented synchronous measure() without the a_measure() coroutine expected by DeepEval's asynchronous mode. The authors estimate that changing the async setting or adding a three-line wrapper would raise Eval@1 from 65.0% to 87.5%.
The 87.5% figure is a repair estimate after failure analysis, not an observed first-run score. It suggests a practical loop for generated evaluators: execute, classify the failure, change one integration, and rerun.
Human preference does not erase the limits
Three experts blindly compared EvalAgent with the two-stage baseline across 40 agent-and-requirement pairs. They preferred EvalAgent 79.5% of the time, tied 10.5%, and chose the baseline 10.0%. Fleiss' kappa was 0.923.
The paper's model-based meta-evaluator matched the human-majority overall winner in 39 of 40 cases. Agreement was lower for individual dimensions, especially plan quality and plan-code alignment. One aggregate judge result should not be treated as ground truth.
AgentEvalBench has 20 agents, all evaluation generators use Claude-family backbones, and the runs start from pre-collected traces. Embodied and multimodal agents are outside its scope. The Apache-licensed Agent-EvalKit repository packages the workflow for Claude Code, Kiro CLI, and Kilo Code. Its README tells users to review and customize what it generates. AWS also published a technical walkthrough using a travel research agent, but that demonstration is not an independent benchmark.
Give the evaluator an acceptance test
Choose two or three metrics tied to failures you have observed. For each one, record the evidence it consumes and the output expected from a passing trace. Add one trace that must fail. Keep extraction in one shared parser.
Run the evaluation from a clean environment before reading the report. An entry-point crash is a rejection. So is a test file that never gets imported or a metric that returns the same score for every input. Compare a small sample with human judgments before scaling an LLM judge.
If models are swapped through api.ish.chat or another gateway, keep the target model, evaluator model, judge, prompt, and library versions in the run record. Any of them can move the score.
Only after the evaluator executes on a passing and failing trace should its numbers influence a release. Eval@1 gives teams a simple way to enforce that order.
