A benchmark starts with a task you understand

AgentArena ships ten core task packs covering small jobs such as repairing typed configuration, fixing a failing arithmetic test, restoring input-validation boundaries, improving structured logging, and adding focused tests. You can also create a task with a natural-language goal, a target repository, and optional expected changed paths.

The generated build, test, and lint checks are only basic repository-health evidence. AgentArena warns that they do not prove the requested product behavior is correct. If expected paths are omitted, the report marks the change scope as unconstrained. That warning belongs in every coding-agent comparison. A green build shows something, but it does not prove the requested behavior works.

The project supports command, test-result, lint, compilation, snapshot, JSON schema, file-content, patch-validation, and other judge types. Its task-pack guide recommends marking essential checks as critical and testing a pack with built-in demo agents before using a real provider.

Build the first pack from a task your team has already solved. Freeze the repository just before the fix, describe the expected behavior without copying the solution, and write judges that would have caught the original defect. A task nobody understands can produce a precise-looking score with no practical meaning.

Fairness means holding the inputs still

AgentArena's fairness rules copy the same repository snapshot into isolated workspaces. Agents receive the same task prompt and task-level environment allowlist, while setup, judges, and teardown stay shared. Preflight results travel with the comparison.

Those controls do not make different tools identical. Authentication state may differ. One command-line agent may expose a detailed event stream while another provides little more than standard output. Token and cost data may be known, estimated, or unavailable. AgentArena preserves those labels instead of turning a missing cost into zero.

The current adapter matrix makes the limits visible. Codex is the supported external path. Claude Code is experimental. Cursor, Gemini CLI, Aider, Copilot, OpenCode, Qwen Code, and several others remain explicit experimental adapters, while Windsurf is blocked. The capability table also says trace richness and cost reporting differ by adapter.

A model name leaves most of the comparison unspecified. The agent version, provider route, reasoning setting, local instructions, installed skills, tool permissions, and repository state can all change the result. AgentArena freezes or records many of those identities. Its reports apply to that configuration.

The score is deliberately local

AgentArena's default practical score gives most weight to status, tests, and critical judges, followed by duration and cost. Failed runs are capped in a low score band. A run that passes overall while failing a critical judge is capped in a middle band. Speed and cost create meaningful separation only after the task is basically complete.

The project's scoring documentation states what the number does not mean. It is not a universal model benchmark, a cross-task absolute ranking, or a vendor-quality claim. It ranks variants inside the current run.

Historical results are kept separate. They are grouped only when task identity, score mode, base agent, provider profile, model, and agent version all match. Change one of those fields and a new history row starts. That avoids a common benchmark mistake: allowing a newly updated tool to inherit victories produced by an older setup.

One clean run is still only a baseline. Repetition matters because agent execution can vary even when the visible task does not. The historical view tracks sample size, success rate, first-pass rate, average score, win rate, median duration, and median cost. First-pass rate is especially useful for workflows where retries consume review time or create noisy diffs.

Run a pilot before building a leaderboard

AgentArena requires Node.js 22 or newer. Its built-in demo path needs no API key or external provider:

npm install --global @agentarena/[email protected]
agentarena init-taskpack --template repo-health --output agentarena-task.yaml
agentarena run --repo . --task agentarena-task.yaml --agents demo-fast,demo-thorough
agentarena ui

For a real comparison, start with one agent and one task. First confirm that the untouched repository passes its checks. After the run, inspect the diff, failed judges, trace, and evidence before looking at the score. Repeat the same configuration enough times to expose flaky behavior. Then change one variable, such as the model or agent, and run the trial again.

Keep the execution boundary in view. AgentArena runs generated code and task-pack commands. Its security decision record describes command allowlists, blocked shell and eval forms, filtered environment variables, and process timeouts. It also documents escape hatches. Treat external task packs as code, review them before running, and keep the local interface bound to loopback unless you have designed a stronger network boundary.

Public suites remain useful for screening models and following progress. Local replay tests whether an apparent advantage survives contact with your repository and your definition of done.

Our earlier guide to proof-oriented vulnerability evaluation explains why a trigger alone is not enough evidence. The same discipline applies here. RepoComplianceBench showed that repository instructions can produce partial compliance that looks better than it is. AgentArena provides a place to encode the checks that distinguish a plausible patch from a useful one.

The best result from a local benchmark is not a permanent winner. It is a task, snapshot, judge set, and evidence trail that can be rerun when the model, harness, permissions, or codebase changes.