title: "SWE-Explore found coding agents recall only 14% to 19% of the lines successful repairs used" slug: "swe-explore-coding-agents-miss-needed-lines" excerpt: "SWE-Explore separates repository reading from patch writing and finds that agents often reach the right file while missing most of the useful lines inside it." category: "Developer Tools" tags:
- "SWE-Explore"
- "coding agents"
- "benchmarks"
- "context retrieval"
- "software engineering" author: "ISH Team" read_time: "7"
A coding agent can fail before it writes any bad code. It may open the right repository, identify a plausible module, and still miss the helper, test, or configuration span that explains the bug. End-to-end repair benchmarks put that failure in the same red box as a bad patch.
SWE-Explore separates reading from repair and measures it directly. Given an issue and a fixed repository snapshot, an explorer returns five ranked code regions. The benchmark compares those file paths and line ranges with evidence extracted from independent agent trajectories that solved the issue.
Its most revealing result is not an overall leaderboard. General coding agents including Claude Code, Codex, OpenHands, Mini-SWE-Agent, and AweAgent had strong file-level hits and put useful results near the top. Their line-level recall still stayed around 0.14 to 0.19. They reached the correct neighborhood, then left most of the useful evidence unread.
That gap matters on private codebases. A pass rate tells a team whether the whole system worked. SWE-Explore asks an earlier question: did the agent assemble enough of the repository to have a fair chance of writing the fix?
What SWE-Explore scores
The benchmark contains 848 issues from 203 open-source repositories in 10 programming languages. Tasks come from SWE-bench Verified, SWE-bench Pro, and SWE-bench Multilingual. On average, a repository has 759 non-test files and about 179,600 lines of non-test source code.
The final patch is not the ground truth. The researchers retain issues with at least two successful repair trajectories, extract explicit read actions, and identify the regions those runs consulted. Editor views, command-line reads, and line-numbered search hits can count. They discard terminal activity that cannot be mapped to an exact file and interval. An LLM-assisted refinement step may promote useful optional reads, and the authors manually audit every refined target.
Core context averages 4.3 files, 4.7 regions, and 1,578 visible lines per issue. Each explorer returns its top five regions. The evaluator measures line precision and recall, file and region hit rates, how soon useful evidence appears, and how much of the line budget goes to relevant context.
One output format lets the study compare BM25, TF-IDF, a lightweight dense retriever, general coding agents, and specialized localization systems without requiring every system to write a patch.
The project repository contains the dataset schema, explorer wrappers, metric code, and restricted-context validation scripts. The team also published the dataset on Hugging Face.
The right file can still be the wrong context
Agentic explorers beat one-shot retrieval in the study. Classical sparse retrieval and the lightweight dense retriever stayed close to random on many metrics. Interactive agents could search, inspect a result, and change direction.
But the agentic methods clustered at an awkward operating point. They were good at finding files and weak at covering the relevant lines inside them. Using the same Mini-SWE-Agent scaffold, GPT-5.4 hit 65.5% of core files but recalled 15.4% of core lines. GPT-5.4-mini hit 64.9% of core files and recalled 18.5% of lines. Sonnet-4.5 hit 53.5% of files and recalled 11.8% of lines.
Changing the model moved the scores without removing the bottleneck. Five general coding agents also produced similar exploration profiles despite different harnesses. More orchestration did not automatically produce broader evidence coverage.
CoSIL was the main exception among specialized localizers. Its iterative code-graph search achieved the highest non-oracle line recall and F1. The paper also warns that returning whole files can inflate recall by adding noise. A good explorer must search broadly enough to catch a dependency, then rank decisive spans early enough to fit the context budget.
Missing evidence hurt more than moderate noise
On a shared subset of 150 issues, each explorer supplied five regions as the only repository context visible to fixed Mini-SWE-Agent patchers. The researchers then checked whether exploration scores tracked repair. Context efficiency had a Pearson correlation of 0.950 with downstream resolve rate across the explorer pool. Recall within a 100-line budget had the strongest rank correlation, with Spearman's rho of 0.845.
These correlations compare the tested explorers; they do not predict every individual repair. A separate degradation experiment helps explain the pattern. On the easier issue subset, resolve rate remained low with partial core evidence, then jumped between 50% and 75% of the core regions. Once at least 75% of core context was present, moderate amounts of irrelevant code did less damage than missing required evidence.
With no core context, adding random repository code reduced resolve rate by 7 to 9 percentage points. Noise still has a cost. A missing necessary span can be worse than some extra code after the necessary set is complete.
Test retrieval on work your team already solved
Start with resolved issues and freeze the repository commit for each one. For every successful investigation, record the files and line ranges that engineers used, including evidence outside the final patch. Then ask each agent or retrieval layer for a ranked region list before editing begins.
Keep the line budget fixed so an entire repository dump cannot pass as good search. Score file hits, line recall, first useful hit, and context efficiency separately. Finally, give only the selected regions to one fixed patcher and run the original tests. That last step shows whether a higher retrieval score survives contact with the repair task.
This replay can sit beside AgentArena's local task packs. It also gives more substance to entity-level diffs from sem. A cleaner diff helps review only after the agent has found enough evidence to change the right entity. The result may also explain some of the time pressure measured by CCBench. An agent that keeps reopening plausible files can spend its task budget before patch quality becomes the main problem.
The score has boundaries
Trajectory-derived ground truth records what successful agents read. It does not prove that every valid solution needs the same lines. SWE-Explore also excludes issues without at least two successful trajectories, so its 848 tasks do not represent all unsolved repository work. The restricted-context repair protocol validates the exploration metrics under controlled conditions. It is not an absolute measure of patch-generation ability.
A file-level hit can look reassuring even when a file has hundreds or thousands of lines. SWE-Explore shows how much evidence can still be missing inside that success. Before paying for a stronger model or adding another orchestration layer, check whether the system surfaces the connected spans that make the issue solvable. Better patch generation cannot use evidence that never entered context.
