A benchmark that begins before the ticket

Active-SWE, a preprint by Haobin Li, Ping Deng, Weizhong Qian, Liang Jiang, Zhenyu Huang, Mouxing Yang, and Xi Peng, builds tasks from merged pull requests in 87 open-source repositories with at least 1,000 GitHub stars. Its extended set contains 1,663 tasks across Python, Go, Rust, PHP, Ruby, JavaScript/TypeScript, Java, and C/C++. The curated main benchmark contains 400 tasks: 300 simple and 100 hard.

The six bug categories cover Logic and Computation, Reference and Data Flow, Data Processing and Type, Domain-specific Workflow, Exception Safety, and State and Lifecycle. For simple tasks, the benchmark removes the original issue report but gives the agent the files touched by the reference patch as its review scope. Hard tasks combine temporally adjacent fixes so that the snapshot contains at least two recorded bugs.

That file scope changes the interpretation. Active-SWE asks the agent to search without a ticket, but not to audit an entire organization. The benchmark already knows which files the historical fix touched. A production tool with no issue report and no review boundary faces a larger search problem.

The authors used Claude Code as a shared scaffold for every evaluated model. Results therefore compare model and benchmark behavior inside one harness, not every commercial coding product in its native configuration.

Finding the right lines is the bottleneck

The main 400-task results are low even before comparing them with the reactive subset. Claude Opus 4.8 has the highest recorded-bug resolved rate at 20.0%. GPT-5.5 reaches 18.5%, Gemini-3.1-Pro 15.8%, and the best open-source result in the table, Kimi-K2.7-Code, reaches 14.8%.

Localization recall tells a similar story. The best value is 29.1% for Claude Opus 4.8, followed by 28.8% for GPT-5.5 and 26.8% for Gemini-3.1-Pro. The paper defines localization by matching edit hunks in the generated patch against the human reference patch. Its analysis finds that correctly located bugs are more likely to be resolved, while noting that some successful patches take a different path from the historical fix.

A "scan my repository" feature has to prove both abilities. A model may repair a defect once shown the failing behavior, relevant file, and expected outcome. That does not establish that it can select the same defect from hundreds of files and competing code smells.

Track discovery separately from repair. Record which files the agent inspected, which suspected defects it proposed, which tests reproduced them, and which fixes passed existing regression tests. One blended success score hides where the workflow failed.

Potential bugs need executable evidence

Active-SWE also scores defects beyond the recorded historical patch. Because no human oracle exists for those claims, the agent must generate tests that fail before its change and pass afterward. A separate judge model checks whether each proposed bug is semantically connected to at least one valid reproducing test.

The paper reports three potential-bug measures. Count is the number of generated tests. TV is satisfied only when every generated test exhibits fail-to-pass behavior. Revealed additionally requires test evidence for every claimed bug.

More tests do not automatically produce a better result.

Claude Opus 4.8 records 81.0% test validity and 75.0% revealed performance in the authors' table, while GLM-5.2 records 79.0% and 69.0%. These percentages do not mean the models resolved three quarters of the benchmark. They belong to the potential-bug track and its evidence conditions, whereas the 20.0% figure measures complete repair of recorded bugs.

The judge introduces another limitation. Qwen3.5-397B supplies semantic associations between generated tests and claimed bugs. Fail-to-pass execution gives the evidence a concrete base, but an LLM still decides whether the test demonstrates the stated defect. Independent replication and human review remain useful before treating a newly discovered issue as real.

The evaluation closes obvious escape routes

The public implementation runs the Recorded and Potential stages in Docker containers with --network none. Model calls pass through a fixed-destination Unix-socket tunnel. Original Git history is replaced with one clean worktree commit, and prompts prohibit issue, pull-request, commit, and upstream-source lookup.

Those controls stop an agent from recovering the missing issue report through Git history or searching for the public pull request. The project's network-isolation document says the Judge stage remains outside this boundary. Its networking should not be confused with the restrictions on bug generation.

The controller produces metrics.json as the authoritative result. A run status of ok only means the pipeline completed; the summary can still contain missing patches, missing tests, or absent judge results. That distinction should carry into internal agent dashboards. Infrastructure success is not repair success.

A safer proactive-review workflow

Teams can borrow the benchmark's shape without deploying an autonomous repair bot. Start with a bounded change set, subsystem, or pull request. Ask the agent to produce suspected defects and reproducing tests before accepting code changes. Run those tests against the unchanged and patched versions, then require the normal regression suite and human review.

Our AgentArena guide explains how to freeze a repository task and preserve judge evidence. The patched-counterfactual guide explains why a trigger alone is not enough to prove a vulnerability. Active-SWE applies the same instinct to general bugs: discovery earns trust through a failing test and a verified repair, not through a confident paragraph.

The reported 33 to 41-point gap begins before code generation. When the issue report disappears, localization becomes the work. Any product promising continuous autonomous maintenance needs to measure that work directly.