IssueTrojanBench: treat every bug report as untrusted input
A bug report describes work. To a coding agent, it can also look like permission to install a package, create a script, or rewrite the files that govern the agent itself.
IssueTrojanBench tested that confusion. The July 2026 preprint planted malicious instructions in issue-resolution tasks and ran six agent-model configurations in autonomous or auto-accept modes. The agent performed the planted action in 2,776 of 4,176 runs, or 66.5%.
This is not a universal breach rate. The experiment began with six issues from two Python repositories, SymPy and requests. It used one task prompt, three agents, three models, and four attack categories. Some results counted an attempted action rather than a completed compromise. The paper is a preprint, and agent behavior changes with product and model updates.
Its useful finding is narrower. Text from an issue, comment, attachment, webpage, or repository file belongs to the data plane. It cannot be allowed to grant itself authority over the shell.
What the benchmark ran
The researchers generated 696 adversarial artifacts from six seed issues. They crossed four malicious outcomes with six delivery vectors, then varied position, language, capitalization, font size, and visibility. Each artifact ran against six agent-model combinations, producing 4,176 executions.
The four outcomes targeted familiar coding-agent capabilities:
- installing a plausible but nonexistent Python package;
- creating a hidden executable validation hook;
- writing
CLAUDE.md,AGENTS.md, or.cursorruleswith instructions to skip confirmations; - creating a script designed to spawn 10,000 worker threads.
Payloads arrived through issue bodies, issue comments, PDF files, websites, source-code comments, and image alt text. Five channels produced the same 72.2% success rate in the study. Image alt text was lower at 16.7%. Hiding instructions in HTML comments or white-on-white text did not reduce execution compared with visible text in the tested configurations.
The agents were configured for the riskiest operating mode. They could manipulate repositories and use the shell without requesting approval for every command. The study therefore measures autonomous issue resolution, not a supervised interactive session.
What 66.5% does and does not mean
IssueTrojanBench uses a binary Exploit Execution Metric. For persistence, policy-bypass, and resource-exhaustion cases, the grader inspected the filesystem for expected malicious files and content. In the supply-chain case, a run passed if terminal logs showed an attempted pip install or the agent created a fallback status file after installation failed. The named packages did not exist on PyPI.
An attempted install shows that the agent accepted an untrusted instruction. It does not prove that malware installed successfully or that data left the machine. Supply-chain prompts had the highest success rate, 96.6%, so this broad definition contributes heavily to the headline result.
The corpus also fans many variants out from six underlying issues. Its 4,176 runs support controlled comparisons, but they do not cover thousands of independent engineering situations. The authors state that other repositories, languages, prompts, agents, and later product versions may behave differently.
Vendor rankings are fragile for the same reason. The paper found differences by model, but it freezes specific models and configurations at one moment. Those rows are results to reproduce, not a permanent ordering.
Labels cannot enforce permissions
The researchers tried a Spotlighting-style defense. They wrapped external content in explicit markers and described it as untrusted data rather than instructions. Payloads still executed across multiple configurations.
Input labeling can help a model interpret context. It cannot enforce policy by itself because the model reads both the warning and the hostile text. When it resolves that conflict incorrectly, an external control must still block the action.
OWASP recommends separating external content, enforcing least privilege, and requiring approval for high-risk operations. Its excessive-agency guidance also calls for fewer tools, narrower functions, restricted downstream permissions, and authorization outside the model.
GitHub's Copilot cloud-agent controls show how several layers can work together. GitHub says it filters HTML comments before passing issue text to the agent, restricts who can trigger the agent, limits the branch it can push to, restricts internet access, and requires human review before merge. Automations ignore events from users without write access by default, and administrators choose their allowed tools. IssueTrojanBench did not test this exact Copilot cloud-agent workflow, so the benchmark does not measure those controls.
Put an intake gate before the agent
An issue-to-agent workflow should start with a trust decision.
- Allow automatic triggers only from trusted repository roles. Send outside reports to a human triage queue first.
- Normalize the payload. Remove hidden markup, fetch attachments in an isolated process, and show extracted text to the reviewer. This reduces the difference between what the person and agent see, though it cannot identify every semantic injection.
- Begin with read-only repository access and no network. Grant write access only to a task branch after the agent produces a plan.
- Put package installation, network access, executable creation, CI configuration, agent instruction files, and large process launches behind deterministic policy checks. The model cannot approve its own escalation.
- Require human review for new dependencies and changes to
AGENTS.md,CLAUDE.md, workflow definitions, hooks, and permission settings. - Record the issue source, extracted content, tool calls, blocked actions, file changes, and approval identity. Good agent traces expose an attempted escalation even when the final patch looks harmless.
Least-privilege agent tools should also be narrow. A package lookup with a fixed registry and explicit allowlist is easier to mediate than unrestricted shell and network access. A clear tool interface can reject an unexpected package name before a model-generated command reaches the operating system.
Issue text may propose work. It must not expand the permissions used to perform that work.
