title: "EarlyEval cut up to 44.1% of input tokens, but it should not set headline scores" slug: "earlyeval-44-percent-input-tokens-not-headline-scores" excerpt: "EarlyEval predicts outcomes from partial agent trajectories to make repeated benchmark runs cheaper, while leaving final published scores to full execution." category: "AI Research" tags:
- "EarlyEval"
- "agent evaluation"
- "SWE-bench"
- "TerminalBench"
- "benchmarking" author: "ISH Team" read_time: "7"
A coding agent applies the right edit halfway through a benchmark run, then spends the remaining steps rerunning tests. Another repeats the same failed command until the clock expires. The final label may already be predictable, but the evaluator keeps paying for the tail.
EarlyEval is designed to stop that spending. It trains two LightGBM classifiers on completed benchmark trajectories, one to predict success and another to predict failure. During a fresh run, both models inspect each partial trajectory. Execution stops when either calibrated confidence score crosses its threshold. If neither one is confident, the agent continues.
At the authors' recommended settings, EarlyEval removed 13% to 26% of execution steps across SWE-bench Verified, TerminalBench, and Toolathlon. Toolathlon input tokens fell 44.1%. On SWE-bench Verified, the method cut 26.0% of steps, 32.7% of input tokens, and 28.7% of output tokens. The measured Pass@1 for each agent moved by 1.1 percentage points on average.
That error is small enough to help with prompt tuning. It can still reverse a leaderboard claim when two systems are separated by less than one point.
Predicting the end from the middle
Training starts with completed trajectories and their binary outcomes. EarlyEval expands every trajectory into prefixes, gives each prefix the final label, and extracts roughly 500 features from three families.
Behavioral features record actions, edits, tests, errors, repeated operations, no-edit streaks, and milestone timing. Text features use TF-IDF and truncated SVD over the task prompt, action history, and environment feedback. For a benchmark with a gold solution, reference features compare the files, symbols, and tests touched so far with that solution.
Separate success and failure predictors create an uncertain region. When both probabilities stay below their thresholds, EarlyEval does nothing. Ambiguous trajectories keep running instead of receiving a forced early verdict.
One example in the paper follows a 45-step OpenHands trajectory. The agent reproduced the bug at step 20 and made its only source edit at step 23. It never changed the patch again, although it continued testing. A predictor that stopped after the correct edit could preserve the result at roughly half the execution length.
Behavior beat an LLM judge
LightGBM can score the feature vector in under a millisecond on one CPU core. The researchers compare it with a direct multilayer perceptron, two logistic-regression variants, and a LoRA-tuned Qwen-0.5B judge.
At a 0.95 threshold on SWE-bench Verified, LightGBM halted 34.8% of trajectories with 95.0% accuracy and saved 26.0% of steps. The Qwen judge halted 18.7% with 90.7% accuracy and saved 17.9% of steps. It also needed a model forward pass at every step, consuming part of the compute that early stopping was supposed to save.
The feature ablation explains why a conventional model worked so well. Removing every behavioral feature reduced coverage from 34.8% to 23.4% and step savings from 26.0% to 16.4%. Removing the reference-solution family left 32.1% coverage and 24.7% step savings. The shape of the agent's work carried more stopping signal than overlap with a gold patch.
TerminalBench and Toolathlon do not release per-task reference solutions, yet EarlyEval can run on both. A tool agent retrying an unchanged error and a shell agent cycling through the same commands expose failure through behavior alone.
Cheaper estimates are still estimates
The separate confidence thresholds for success and failure control the tradeoff. On SWE-bench Verified, lowering the threshold from 0.95 to 0.75 raised step savings from 26.0% to 63.4%. Mean absolute Pass@1 deviation rose from 1.1 to 4.1 percentage points.
Success and failure predictions do not generalize equally across benchmarks. Success precision stayed between 88.3% and 93.9% on SWE-bench Verified across the tested thresholds. It weakened on TerminalBench and became effectively inactive at higher Toolathlon thresholds. The failure model was steadier: it reached 96.7% at the selected SWE-bench setting, 89.4% to 96.6% in the selected TerminalBench settings, and 96.6% at Toolathlon's selected point.
Early-stopped rankings had Spearman correlations from 0.959 to 0.994 with the full-run leaderboards. Exact placement was shakier. On SWE-bench Verified, 81% of agents kept their rank. The figure was 59% in TerminalBench's no-same-model split and 70% in its no-same-scaffold split and Toolathlon.
These results support quick directional comparisons, not equivalence with executed outcomes. The paper says canonical, citable scores should still come from complete runs.
What the open release does and does not include
The EarlyEval repository is MIT licensed. It contains feature extraction, training, policy application, reports, configurations, and a smoke-test fixture made from nine mini-SWE-agent runs. The full reproduction also requires raw trajectory tables and generated model artifacts that are not committed. The maintainers call it a code-only release, so a clone alone will not rebuild every paper table.
Historical data is another entry cost. The study used more than 21,000 trajectories: 7,805 from 16 SWE-bench Verified agents, 6,757 from 37 TerminalBench configurations, and 7,116 from 22 Toolathlon models. A new benchmark with no completed runs cannot train the predictor.
An established internal suite can adopt the same pattern:
- Keep full trajectories with their final execution labels.
- Split data by task and hold out complete agent configurations, rather than mixing prefixes from the same run across training and evaluation.
- Calibrate success and failure independently on the target benchmark.
- Check estimated scores, rankings, and resource savings against a full-run control.
- Stop early during prompt and scaffold iteration, then run finalists to completion.
This fits the cost problems measured in Badger Code and CCBench. Those studies show where agent budgets disappear. EarlyEval tests whether another step is still likely to change the measured outcome.
Full-run controls also protect against distribution changes. Active-SWE found large performance drops when issue reports disappeared, while SWE-Bench 5G separated diagnosis from successful repair. EarlyEval learns regularities in earlier trajectories. A new scaffold, task format, or failure mode may alter them.
Use EarlyEval as an internal instrument. It can tell a team sooner that a change probably helped or hurt, making more iterations affordable. Final evidence still comes from completing the task and executing the benchmark's verifier.
