What the tool actually changes

LemonCrow runs locally alongside Claude Code, Codex, opencode, and LemonCode. It builds a local code graph, supports exact-range reads, limits tool output, and maintains durable memory. Instead of repeatedly sending broad repository context to a model, it tries to retrieve the smaller slice needed for the current step.

That targets a familiar source of waste. Coding agents often reread files, command output, and repository structure. Better retrieval can reduce the churn, but it introduces its own risks: an index can be stale, a symbol can be missed, or an integration can apply weaker checks. LemonCrow's documentation says session-close verification is enforced in Claude Code and advisory in other hosts, a meaningful distinction when comparing setups.

What Terminal-Bench measured

The project's methodology describes a paired Terminal-Bench 2.1 test across 89 tasks, with five repetitions per task. Each arm therefore contains 445 trials.

The reported totals are:

• Correct trials: 351/445 for both configurations, or 78.9%
• Fresh input tokens: 182,000 with LemonCrow versus 12.87 million for the baseline
• Normalized cost: $61.98 versus $73.75, a 16.0% reduction
• Cost coverage: 86 of 89 tasks had priceable trajectories on both sides

LemonCrow links the Harbor job and publishes raw benchmark artifacts. Public artifacts do not prove that every team will see the same result, but they let readers inspect more than an isolated chart.

The benchmark version is part of the result. Terminal-Bench 2.1 fixed problems in 28 of its 89 tasks from version 2.0, including external dependency drift, resource mismatches, and misspecification, according to the official release notes. “Terminal-Bench” without a version is not enough information for a comparison.

The cache math behind the smaller saving

Fresh input is only one entry on a current model bill. Cache writes, cache reads, and output tokens may all have different prices. A context layer can replace expensive fresh input with discounted cached reads, but discounted does not mean free. It can also incur cache-write costs before later reads pay off.

LemonCrow normalizes both arms to a one-hour cache-write rate. The baseline run actually used the cheaper five-minute cache-write tier, so the maintainers repriced it for the comparison. The normalized total answers what both configurations would cost under the same cache-write assumption. It is not the exact invoice for either recorded run.

There is also a denominator change. Correctness covers all 89 tasks. Cost covers the 86 tasks with priceable trajectories in both arms. Combining those figures into one all-purpose claim would blur two different measurements.

This is why “tokens saved” is incomplete unless a report separates fresh input, cache writes, cache reads, output, and the price table applied to each. Our guide to agent context budgets explains how retrieval, caching, trimming, and compaction act on different parts of the context pipeline.

The tempting comparison that does not work

The benchmark page also lists a standalone Opus 5 row with an 86-task cost of $38.68. LemonCrow's maintainers explicitly say not to subtract that figure from the Opus 4.8 baseline. It uses its own model-tier billing and a different set of excluded tasks, and there is no matched baseline arm. The row describes one run; it does not isolate the effect of LemonCrow.

Results on the SWE-bench variants reinforce that caution. On a fixed 50-task SWE-bench Verified sample with five repetitions, LemonCrow reports 232/250 correct trials versus 202/250, with cost down 29.5%. On SWE-bench Lite it reports 48/50 versus 49/50, a two-percentage-point correctness loss, while cost falls 11.7%. On SWE-bench Pro it reports 45/50 versus 44/50 and 21.5% lower total cost, but three tasks with tied correctness were more expensive with LemonCrow.

A one-repetition current-build spot check dated July 30, 2026, reached 38/50 on the same Verified sample, compared with 80.8% for the older five-repetition baseline. The project labels the check noisy and does not call it a regression. That restraint is warranted: different builds and repetition counts do not support a clean causal claim.

How to test a context layer fairly

A purchasing or adoption decision needs a paired test on the repositories and tasks that matter to your team.

1. Pin the dataset revision, harness, tool versions, model, provider route, timeout, and machine resources.
2. Use several repetitions per task. Agent outcomes vary, and a single pass can exaggerate a win or a failure.
3. Record fresh input, cache writes, cache reads, output tokens, actual billed dollars, and a separately defined normalized cost.
4. Keep task-level paired deltas and list every exclusion. A lower total can conceal an expensive regression on a critical job.
5. Log the model that was actually served. The RuBench fallback incident showed why the requested model and the served model cannot be assumed identical. If you test through an OpenAI-compatible gateway such as api.ish.chat, lock the route and retain the response metadata.
6. Review privacy settings before using the tool in a company repository. LemonCrow says anonymous telemetry is enabled by default and includes aggregate counts, durations, dollar estimates, and a hashed installation key, but not source code, prompts, paths, or symbol names. Disable it with lc telemetry remote off, DO_NOT_TRACK=1, or LEMONCROW_TELEMETRY=off when policy requires it.

Run that test long enough to price cache reuse honestly. Adopt the tool if the real bill falls without reducing the pass rate on work you care about. If only the fresh-token percentage looks good, keep measuring.