title: "SWE-Bench 5G models diagnosed over 91% of bugs but resolved only 10% to 30%" slug: "swe-bench-5g-diagnosis-resolution-gap" excerpt: "A telecom benchmark shows that recognizing a bug is much easier than producing a patch that compiles, applies cleanly, and follows the relevant protocol specification." category: "AI Research" tags:
- "SWE-Bench 5G"
- "coding agents"
- "telecommunications"
- "benchmarks"
- "3GPP" author: "ISH Team" read_time: "7"
Knowing what is wrong with code is not the same as fixing it. SWE-Bench 5G makes the distance visible: all four tested language models diagnosed more than 91% of its telecom bugs, while multi-turn resolve rates ranged from 10% to 30%.
The benchmark takes coding agents into 5G core network software. Correctness may depend on protocol state, distributed network functions, strict Go or C types, and a precise 3GPP clause. That makes it a useful counterpoint to general Python repair scores. An agent can explain a bug correctly and still fail because its patch does not apply, does not compile, misses another dereference, or implements the wrong protocol behavior.
For an engineering team, a high diagnosis rate shows that a model can often understand the issue. It does not show that the agent around that model can turn the explanation into a production-quality change.
What is inside SWE-Bench 5G
The researchers mined 555 candidate issues and merged pull requests from 33 repositories in three open-source 5G core projects: free5GC, Open5GS, and Magma. Validation left 210 instances: 128 from free5GC, 58 from Open5GS, and 24 from Magma.
Tasks span Go, C, and Python across seven network functions. The set includes 89 nil or null pointer bugs, 42 crashes, 35 missing validations, 27 logic errors, and 17 concurrency bugs. Each task runs in a Docker environment at the buggy commit. Pass-to-pass tests protect existing behavior, while fail-to-pass tests check the target bug.
Telecom code can be difficult to exercise in isolation. Some functions require SCTP connections, database sessions, or state shared across network functions. SWE-Bench 5G uses two test strategies: 68 instances call the function directly, and 142 use diff-based intent checks to verify that an expected fix pattern appears in the source.
The second strategy makes many tasks testable, but narrows the meaning of a pass. A source-pattern check can confirm that an agent added a required guard or structural change. It cannot reproduce every runtime behavior of a live 5G core. The paper states this testing compromise directly.
The Hugging Face dataset includes task metadata and the containerized evaluation setup.
Diagnosis was the top of a much steeper funnel
The evaluation used Qwen3.5-Flash, Kimi-128k, GPT-4.1, and Claude Sonnet 4. In multi-turn mode, an agent had up to five attempts. After each patch, the harness applied it, compiled the code, ran tests, and returned failures for another revision.
Claude Sonnet 4 resolved 30.0% of the 210 tasks. GPT-4.1 resolved 20.0%. Qwen3.5-Flash and Kimi-128k each resolved 10.0%. Their patch application rates were much higher: 79.5% for Claude Sonnet 4, 67.1% for GPT-4.1, 57.1% for Qwen3.5-Flash, and 51.4% for Kimi-128k.
Applied patches still failed for specific reasons. Incomplete fixes accounted for 72 to 80 failures per model. An edit might repair one dereference while missing another site or edge case. Patch formatting was another major loss, especially for Qwen and Kimi, with 75 and 84 failures. The harness required SEARCH blocks to match source whitespace and formatting exactly.
That exact-match mechanism is partly a benchmark constraint. Native file editing or fuzzy patch application could rescue some structurally correct attempts without changing the model's telecom knowledge. The authors identify more robust patch application as a likely benefit for weaker models.
Qwen3.5-Flash makes the value of iteration especially clear. It resolved no tasks in a single response despite diagnosing more than 90% of the bugs. With up to five turns, patch application rose from 34.3% to 57.1%, and resolution reached 10.0%. Compiler and test feedback turned some correct ideas into valid edits.
SWE-Explore found a similar hidden break earlier in the workflow: agents often reached the correct file but missed most of the useful lines. SWE-Bench 5G follows the rest of the path. The agent must express a complete edit, apply it, compile it, and satisfy the tests. One end score hides each handoff.
3GPP context helped some bugs and did nothing for others
Many issues cite a specific 3GPP Technical Specification. For 50 tasks, the researchers created concise Markdown references averaging 350 tokens. Each document summarized the relevant protocol semantics and connected them to the task without revealing the fix. Claude Sonnet 4 then ran in paired conditions with and without that reference.
Overall resolution rose from 24.0% to 30.0%, while average token use increased 12%. The split by bug type matters more than the average. Six groups of generic nil or crash bugs received no improvement. Policy authorization gained 16.7 percentage points, policy control gained 25.0 points, and session management gained 20.0 points. A system architecture group saw no change.
The references helped when correct behavior depended on field optionality or a specification-defined value range. They did not help with ordinary defensive checks. This suggests a better way to use task skills: retrieve a short authoritative passage when the issue turns on that passage, instead of attaching a broad manual to every task.
The result also sharpens the trust question in TRUSS's controlled skill execution. Relevant documentation can improve a task, but it still changes agent behavior and should be tested through the same patch and verification loop.
What a telecom team can measure
Track diagnosis, patch application, compilation, and test resolution as separate stages. Pin each real issue to an exact commit. Its check should fail on the buggy version and pass after the reference fix.
Retrieve specification passages by clause, then run a no-passage control. Repeat the same task in single-turn and multi-turn modes. The difference shows how much compiler and test feedback contributes.
Failures also need an owner. Record whether the model misunderstood the protocol, the edit transport rejected a reasonable change, or the test harness could not observe the intended behavior. When direct runtime tests are impractical, treat source-pattern checks as partial evidence rather than a substitute for deployment testing.
This accounting matters under limits like those in CCBench. More turns can repair format and compiler failures, but each turn consumes time and tokens. The feedback loop has to help before the task budget runs out.
SWE-Bench 5G does not establish that current agents understand a live carrier network. It draws from three open-source cores, most tasks are easy or medium, and 142 of 210 use diff-based checks. The measured gap is narrower: diagnosis exceeded 91% for every model, while multi-turn resolution stopped between 10% and 30%. That is the distance between recognizing a telecom bug and producing a patch the harness can accept.
