NIKA's best tested agent detected 89% of network incidents but found only 55.3% of root causes
An application is slow and the network dashboard is red. Confirming that something is broken is the easy part. An on-call agent has to choose useful probes, read the right counters, tell a service failure from a routing failure, and locate the cause without making the incident worse.
Many agent benchmarks turn that job into a question with one expected answer. NIKA, an open benchmark built by SANDS Lab and university researchers, gives the agent a live emulated network instead. A known fault is injected, diagnostic tools are exposed, and the agent investigates before submitting its diagnosis.
Results from the original NIKA paper show why the distinction matters. On a 150-incident evaluation subset, the strongest setup tested, a ReAct-style agent using GPT-5, detected abnormal network state with 89.0% accuracy. Localization reached 68.7%. Root-cause accuracy was 55.3%.
Detection is not diagnosis. That gap should concern anyone evaluating an agent for operational work, even if the system under test has nothing to do with networking.
A real network sits behind the prompt
NIKA pairs a library of injectable incidents with an orchestrator. It can deploy data-center, campus, ISP, SDN, overlay, and Kubernetes networking scenarios with emulation backends including Kathará and Containerlab. The current repository catalog contains 75 registered failure types and 1,098 working-matrix cases. The working matrix is development material; it is not the frozen leaderboard suite.
During an incident, the agent can call network diagnostics, switch command lines, and telemetry through MCP servers. The access layer records tool inputs, outputs, timestamps, and snapshots of the actual network state. The paper describes more than 30 tools covering active probes, packet capture, routing configuration, counters, and InfluxDB queries.
That environment tests decisions a static questionnaire misses. Does the agent begin with reachability or chase the first scary log? If ping succeeds, does it inspect the application layer? Does it identify the broken interface, or end with a vague claim that the link is unstable?
The final answer cannot tell the whole story. A lucky label can follow a weak investigation. A missed label may still contain one genuinely useful diagnostic step.
The researchers found that agents often stopped at shallow, connectivity-focused explanations when resource-contention evidence was available. In the paper's category analysis, GPT-5 localization was 97% for link failures and 58% for resource contention. Persistent CRC errors, queue drops, and asymmetric loss did not reliably lead the agent from the visible symptom to the underlying cause.
Version 0.2.0 puts trajectories beside scores
NIKA's frozen 0.2.0 release has 84 development cases and 85 test cases across 29 scenario IDs. Its 169 executable rows cover all 75 registered failures and include 18 healthy baselines. Official runs repeat every case three times.
The release uses rule-based root-cause-analysis F1 as its primary leaderboard metric. Operators may run an optional LLM judge for local analysis, but that judgment does not count toward an official submission. Generated commentary can help a researcher inspect a run without letting another model's preferences determine leaderboard placement.
The September update also introduced nika leaderboard submit. The submission guide sends scores through a GitHub pull request and trajectories through a Hugging Face dataset pull request. A package includes release provenance, run configuration, aggregate results, and per-trial evidence.
The trajectory package makes a lucky score harder to hide. Reviewers can see which tools the agent called and whether the collected evidence supports its conclusion. Failures become material for improving the system too. A team can sort them into wasted calls, early stopping, topology confusion, or fixation on a single layer, then change the harness and rerun the same frozen cases.
Ecdysis uses a related idea: repair an agent runtime from recurring evidence, not one memorable failure. Code Assistant's formatter handling shows the same concern at a smaller scale. A model's next move depends on what its harness accurately remembers.
Keep the paper result separate from the new suite
The 89.0%, 68.7%, and 55.3% results belong to the December 2025 paper, not the 0.2.0 leaderboard suite. The researchers evaluated three model backends on a 150-incident subset. They ran each model twice, with faulty devices and components randomly selected on each run. Every agent used the same two-step LangGraph workflow.
Within that setup, GPT-5 averaged 359.2 seconds and 28.4 tool calls per incident. GPT-5-mini averaged 242.6 seconds and 15.0 calls. This comparison isolates model backends within one harness. It does not establish which model will work best with another tool set, architecture, network, or budget. The newer frozen suite also has a different case mix and scoring rule.
Emulation sets another limit. NIKA can preserve causal relationships while scaling down traffic and link speeds, but the authors say it cannot reproduce every fault that depends on high-speed hardware. The current benchmark evaluates diagnosis, not remediation. It offers no evidence that an agent can safely apply a fix on a production network.
Containment still deserves attention. Agents can invoke consequential diagnostic and service-management tools. NIKA documents isolated sandboxes and controlled access policies, but a benchmark operator should use disposable infrastructure, narrowly scoped credentials, and the MCP-server allowlist declared by the frozen release.
How to test an on-call agent
Begin with a frozen case set and save every run artifact. Score detection, localization, and root cause separately. Otherwise, a strong anomaly detector may conceal an agent that cannot find the faulty component.
Healthy cases matter. An agent that always announces an incident can look competent when every test contains a fault. Repeat cases because one successful run says little about stability. Track elapsed time, tool calls, token use, and operational side effects alongside accuracy.
Then read the trajectories. Check whether evidence came before the conclusion and whether the agent crossed network and application layers. Flag commands that changed state and note where the investigation stopped. Compare recurring patterns across successful and failed runs before rewriting the prompt or adding another tool.
NIKA is not a certificate for production autonomy. Its practical value is the chance to watch an operational agent investigate inside a controlled network. A score records the result; the trajectory preserves the evidence needed to decide whether the process was trustworthy.
