A vulnerability proof needs a patched counterfactual, not just a trigger
A proof of concept runs against a vulnerable build and prints an alarming marker. The container exits successfully, so the dashboard records a reproduced vulnerability. That result may prove very little.
Run the same test against the patched build. If it produces the same signal after the vulnerability has been fixed, the signal was not specific to the vulnerability. It may be ordinary application behaviour, a brittle string match, a designed-in crash, or a bad oracle judging the run.
A new pre-registered reproducibility study applies this standard to artifacts from LLM and agent-driven vulnerability research. Its results are exploratory, drawn from a limited paper sample and one 102-case anchor corpus. Runnability, signal production and semantic confirmation are separate claims.
Public code is only the first rung
The study searched work published from 2023 through 2026 and formed a 104-paper consensus corpus. Fifty-nine papers, or 56.7 percent, had a publicly reachable primary artifact. The researchers executed a stratified sample of 18 paper-level artifacts and separately audited all 102 cases from the anchor vulnerability-validation benchmark.
Only 10 of the 18 paper artifacts completed their declared workflow from a clean snapshot, which the study calls R0. One more ran after environment-only repair, or R1, bringing the total to 11 of 18. R1 could install declared dependencies or make small compatibility adjustments. It could not change the exploit, oracle, or claimed post-condition.
Availability answers one narrow question: can someone obtain the artifact? It says nothing about whether the workflow runs, whether the output matches the paper, or whether the artifact proves the claimed vulnerability.
Research policy already makes this distinction. The ACM artifact-review framework separates availability, evaluated functionality and independently validated results. The ACM CCS 2026 artifact guidance also distinguishes an available artifact from one that is documented, complete, exercisable and useful for producing the paper's outcomes.
A trigger is a candidate signal
The study uses an evidence ladder. G1 is a candidate signal such as a crash, marker, or status change. G2 asks whether that signal matches the CVE-specific post-condition. G3a adds a benign negative control, where the signal should be absent on harmless input. G3b adds the patched counterfactual, where the signal should be absent after the fix.
Strict confirmation requires the post-condition and both controls. Looking for a success string in a log cannot meet that standard.
The extra checks changed the verdicts. Among 30 signal-producing cases with patched-counterfactual verdicts, 20 still produced the claimed signal on the patched build. Seven of 19 matched negative controls also triggered on benign input. Across the audited oracle matrix, artifact-embedded oracles had 60 percent sensitivity and 45 percent specificity.
Those rates describe this corpus. They are not population estimates for all AI security research. The paper says broader confirmatory statistics must wait for a multi-paper case-level pool. The present evidence establishes a narrower point: an oracle can print "triggered" without distinguishing a vulnerable condition from patched or benign conditions.
Sometimes the script names the wrong CVE
The anchor corpus exposed a more basic problem. In 58 of 102 cases, the CVE identifier inside a script differed from the CVE named by its directory. A runner can execute perfectly and still test the wrong target.
Automation often trusts directory names, metadata files, or expected marker strings. When the script points elsewhere, a green result confirms only that the pipeline followed inconsistent instructions.
The pattern is familiar outside security. A coding agent can edit the wrong package and still produce a clean diff. A benchmark can grade the intended task while its process runs a fallback model. Security artifacts make the consequence obvious because the output claims to reproduce one specific flaw.
Before execution, bind four identities together: the declared CVE, vulnerable dependency and version, patched dependency and version, and the oracle's post-condition. If they disagree, stop before interpreting the signal.
Build the counterfactual into the pipeline
A defensible pipeline should produce a small verdict table for every case:
| Build and input | Expected signal |
|---|---|
| Vulnerable build, exploit input | Present |
| Vulnerable build, benign input | Absent |
| Patched build, exploit input | Absent |
| Patched build, benign input | Absent |
The first row says only that something happened. The other rows test whether the signal depends on both the vulnerable version and the exploit input.
This table helps diagnose failures. If both builds trigger, inspect the oracle and the claimed fix. If benign input triggers, the detector may be matching ordinary output. If the vulnerable build never triggers, check the environment, target version and execution path before rewriting the exploit. Keep environment repair separate from semantic changes so a successful rerun does not hide what changed.
The study ran proof-of-concept artifacts inside isolated Docker containers with privileged mode disabled, resource limits, timeouts and per-case network policies. Traffic stayed inside containerized targets owned by the study. Validation work should follow that shape: use owned test environments, pin versions, deny unnecessary host access and record every repair.
For a service built through api.ish.chat, store each evidence level separately instead of reducing the run to one boolean. Preserve vulnerable and patched image digests, input class, raw observation, CVE-specific post-condition and final adjudication. In ish.chat, another model can challenge the oracle, but controlled execution should decide the verdict.
Our article on fail-open agent boundaries covered controls that silently stop enforcing policy. A false-positive vulnerability oracle has the same practical defect. It reports success when its test cannot distinguish the claimed flaw from ordinary behaviour.
Use statuses that preserve the evidence
Security dashboards often jump from "ran" to "reproduced." A better sequence is: available, runnable at R0, runnable after R1 repair, candidate signal, post-condition matched, negative control clean, patched counterfactual clean, and strictly confirmed.
Those labels preserve what each experiment established and make missing evidence visible. A case without a buildable patch is neither confirmed nor disproved. Its status is bounded by the strongest check that ran.
An artifact that survives benign input and stops triggering on the patched build has evidence for a stronger claim. If it only prints "triggered," the correct status is "candidate signal." The next test is already defined.
