A 46-task benchmark gives coding agents credit for progress before they finish
Most agent benchmarks ask a simple question: did the system produce the required final artifact? That is a useful test, but it becomes blunt when a task requires an hour of terminal work, a sequence of dependent fixes, and recovery from a bad early choice. A failed final check can hide whether the agent made substantial progress, stalled halfway through, or spent its time doing unrelated work.
Long-Horizon-Terminal-Bench, or LHTB, is built for that gap. It contains 46 containerized terminal tasks across nine categories, including software and reverse engineering, scientific computing, research reproduction, multimodal analysis, interactive games, and professional workflows. Each task is decomposed into fine-grained graded subtasks, so the evaluator can award partial credit instead of treating every incomplete run as identical.
The idea is straightforward. If an agent correctly sets up an environment, repairs part of a pipeline, and produces a valid intermediate artifact, that work should register. The final task can still fail. A measurement that preserves intermediate evidence is more useful for diagnosing the failure and for comparing two agents that both miss the finish line.
Long work needs more than a finish-line metric
The LHTB paper reports that, across its tested agents, a run averaged 231 episodes, 9.9 million tokens, and 85.3 minutes. Its strongest reported configuration passed 15.2% of tasks at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0. The corresponding mean rates across the evaluated models were 4.3% and 1.7%.
Those figures are not a general ranking of all AI systems. They describe a particular benchmark, harness, budget, and model set. They do make a practical point: an agent that looks competent on a short repository edit may still have trouble sustaining a plan, keeping state, and debugging for an extended terminal session.
Binary scoring has a second problem. It can make two different failures look the same. One agent may make no useful move. Another may build a correct dependency graph, run the right experiment, and fail during a final serialization step. If both get zero, a developer cannot tell whether the next intervention should be a better plan, a more reliable tool call, a checkpoint, or an ordinary bug fix.
LHTB's task layout keeps the public instruction, environment, and task metadata available while withholding the verifier and oracle solution. The public task dataset describes the common layout: task metadata, an agent-facing instruction, and a Docker environment. Hidden tests and reference solutions are not released with the tasks. This is intended to reduce shortcutting and benchmark contamination.
Partial credit is useful only when it tracks real work
Dense grading can be gamed if the substeps reward activity instead of evidence. LHTB's design tries to avoid that by using a reference solution or simulation engine and fine-grained checks. The project repository says it uses hidden, rebuild-from-artifact verifiers. A status message or a claimed milestone does not count by itself. The LHTB repository is explicit about that distinction.
That is the part worth borrowing for ordinary agent evaluation. Track a milestone only when an independent check can reproduce it. For a coding task, that might mean a clean build plus a behavior test. For a data workflow, it could mean regenerating a result from recorded inputs. For a deployment task, use an endpoint check or an observable health signal rather than the agent's final summary.
The result does not need to be a massive benchmark. A team can break one recurring workflow into checkpoints that matter: reproduce the failure, identify the relevant interface, make the change, run a targeted test, run a broader test, and inspect the deployment. Each checkpoint should say what proves it. The habit makes failures legible and gives a later run somewhere safe to resume.
The horizon changes the failure mode
Short tasks mostly test whether an agent can interpret an instruction and write a local change. Long tasks add bookkeeping. The agent has to remember which assumption it made, distinguish an old error from a new one, decide when a command changed state, and avoid endlessly revisiting a dead end. More context can help, but a long transcript can also preserve obsolete assumptions.
This is why checkpoints are more than progress reporting. They are controlled state resets. A useful checkpoint records the current artifact, the command that verifies it, unresolved questions, and the next decision. A fresh agent or reviewer can inspect that record without trusting every inference in the preceding transcript.
The benchmark's nine categories also matter. A task that reproduces research, audits a scientific computation, migrates a framework, or plays an interactive game has a different surface form. All still require sustained interaction with an environment that pushes back. A practical evaluation suite should include the kinds of work a team actually delegates, rather than relying on one neat coding puzzle.
Use partial scores for diagnosis, not marketing
A partial score can be tempting to treat as a softer success rate. It is more valuable as a map of where agents stop. If a system frequently reaches build and test milestones but fails at integration, invest in interfaces and end-to-end fixtures. If it drifts before producing a reproducible intermediate artifact, reduce the task size or add explicit checkpoint prompts. If it passes visible tests and fails hidden verification, look for a shared assumption between the implementation and the test.
The paper's authors describe LHTB as a way to capture how far an agent progresses through open-ended workflows. That is a narrower and more useful claim than declaring an agent broadly capable. The benchmark still has limits: 46 tasks cannot represent every production environment, its reward design reflects its authors' choices, and hidden verifiers make independent inspection harder. Those are normal tradeoffs for a benchmark aimed at resistance to shortcuts.
For teams using agents in terminal-heavy work, the immediate lesson is modest. Do not wait for a final pass or failure to learn what happened. Define a few independently verifiable milestones, save the evidence, and inspect the point where progress stops. That turns a long agent run from a black box into something an engineer can debug.
