Coding-agent evaluations: grade the full trajectory
An AI coding agent can produce a plausible patch and still fail the task. It may edit the wrong file, use an unsafe command, work around a test instead of fixing the defect, or leave the repository in a state that only passes locally. A final diff is important evidence, but it is not the whole run.
Coding-agent evaluation needs a trajectory: the record of a trial's task input, starting repository state, tool calls, command output, file changes, test results, and final handoff. It distinguishes a patch that happened to look right from an agent that used the expected workflow and reached a verifiable result.
The goal is not to score every keystroke. It is to make a release decision with evidence that reflects the job you want the agent to do.
Define a task as an executable contract
Start with a small task that a developer could recognize as complete. Specify the repository revision, files or service fixtures available to the agent, allowed tools, time or turn limit, and success criteria. Use a clean checkout or a disposable environment for every trial.
The success criteria should be executable where possible. A bug-fix task might require a focused regression test, the full relevant test suite, a lint or type check, and a diff that does not alter a protected file. A documentation task may require a valid link check and a statement that matches a source fixture. A deployment task needs a simulated target and an assertion that it did not touch production.
Anthropic describes an agent evaluation as a task, repeated trials, one or more graders, and a transcript or trajectory. That separation is useful in a repository too. The task says what should happen. A trial captures one stochastic attempt. Graders decide which conditions were met. The transcript supplies evidence when a score needs review.
Avoid a task that embeds the preferred implementation. "Fix the parser so malformed headers return a typed error" leaves room for a sound solution. "Add a helper named parseHeaderV2 and edit line 81" mostly tests whether the agent copied an instruction. The tests should define observable behavior, not the author’s first patch.
Grade the environment and the result
Use several narrow graders instead of one broad quality score. Each should answer a question that can be checked consistently.
| Grader | Evidence | Example failure |
|---|---|---|
| Outcome | tests, API response, generated artifact | required behavior still fails |
| Repository state | diff, status, protected-path policy | agent changes a lockfile without need |
| Tool behavior | command and tool trace | agent uses an unapproved write path |
| Safety | sandbox logs, mock service audit | agent attempts a production action |
| Handoff | final report and saved artifacts | result cannot be reproduced |
Outcome graders deserve the most weight. A repository test suite, compiler, schema validator, or controlled integration test can check behavior without asking another model to judge it. Use exact assertions for invariants such as exit status, files created, or response fields.
Some requirements are semantic. A code review agent may need to identify the security consequence of a patch, while a support agent may need to explain a policy exception accurately. For those cases, use a rubric-based grader with a narrow question and named evidence. Keep its output separate from deterministic checks. A model judge that says a patch "looks correct" should not override a failing regression test.
Keep the trajectory inspectable
Record enough detail to reproduce a surprising score: task ID, repository commit, agent and model configuration, prompt version, tool definitions, tool inputs and outputs, commands, changed files, timestamps, test results, and grader versions. Do not put secrets or raw customer data into the trace. Redact them before storage or replace them with stable references.
The agent observability guide covers the mechanics of tracing requests and tools. Evaluation adds a different question: did this particular sequence satisfy a declared contract? A trace without a task and grader is a log. A score without a trace is difficult to debug.
Review failures by category. If an agent often opens the right file but stops before tests, the issue may be tool instructions or time budget. If it passes tests while changing unrelated code, tighten the repository-state grader. If a test fails only after an environment reset, the task fixture may be missing a dependency. The trajectory tells you where to investigate.
Run repeated trials and compare like with like
Agent outputs vary. One successful run is a demonstration, not a rate. Run each task more than once under the same conditions, then report the number of successful trials and the total, such as 8/10, alongside the task definition and model configuration.
Keep comparisons fair. Hold the repository revision, tools, permissions, execution environment, task prompt, and grading code constant. Changing a model and its tool schema at the same time makes the result hard to interpret. If one harness provides browser access, a large context window, or a prepopulated cache and another does not, you are measuring the systems together, not only the models.
OpenAI's July 2026 analysis of coding benchmarks is a useful warning. It found task-definition and test issues that can make a benchmark give misleading capability signals. Treat an unexpected score as a reason to inspect the task, hidden assumptions, and grading code. Do not assume the model is wrong because a benchmark says so, or capable because it passed one.
Version the eval set just like production code. A result should identify the task revision, fixture image, prompt, grader, and harness. When a task changes, rerun the baseline or label the comparison as a new series. This also makes it possible to retire tasks that agents have memorized or that no longer resemble current work.
Build a small CI gate first
Begin with five to twenty representative tasks, not a giant benchmark. Include the routine work your team wants to delegate and a few known failure modes: a misleading test name, an unavailable service, a request that must stop for approval, and a task where a neat-looking patch is incomplete.
Run the fast deterministic subset on every agent, prompt, tool, or harness change. Schedule the slower integration tasks separately. Keep failed trajectories as regression cases after removing sensitive data. A fixed bug should become a test that prevents the same failure from returning.
For parallel coding work, isolate each trial in its own checkout and runtime resources. The Git worktrees playbook explains the Git side of that isolation. A clean worktree does not sandbox a process, so the task still needs scoped credentials and a controlled test target.
ISH chat can help compare model responses on a fixed task, while the ISH API dashboard gives teams a place to monitor API usage during repeatable runs. The useful comparison is the one with the same contract, environment, and grader for every trial.
End each evaluation report with the evidence a reviewer needs: success rate, failure categories, a small sample of trajectories, changed task or grader versions, and the next decision. That leaves room for judgment without turning a single benchmark number into a release verdict.
