title: "TRUSS tests generated agent skills before letting them touch the real system" slug: "truss-shadow-runs-generated-agent-skills" excerpt: "TRUSS combines static inspection with controlled shadow execution, showing why a skill that reads safely can still behave badly once an agent runs it." category: "AI Safety" tags:
- "TRUSS"
- "agent skills"
- "runtime verification"
- "coding agents"
- "AI safety" author: "ISH Team" read_time: "7"
An agent skill can look perfectly harmless in a text editor. The workflow makes sense. The documentation checks out. A bundled helper script appears to do what the instructions say. But none of that establishes what an agent will actually do after the skill is loaded, tools become available, and a plausible request arrives.
That gap between an artifact and the behavior it produces is the subject of TRUSS, a framework described in an August 18, 2026 preprint by researchers at Huazhong University of Science and Technology and Nanyang Technological University. Its central idea is refreshingly practical: generated skills should not earn trust from a prose review alone. A shadow agent should run them in a controlled environment where every requested action can be checked and recorded.
On all 187 tasks in SkillGenBench, TRUSS raised effectiveness from 17.11% with no skills to 52.94%. Its benchmark Security rate rose from 50.80% to 100.00%. This does not mean TRUSS can certify an arbitrary skill for the open internet. It means runtime evidence helped on both halves of a problem that a single success score cannot capture: did the skill improve the task, and what did it try to do in the process?
A skill is more than an instruction file
Agent skills package procedures, reference material, and sometimes executable resources. A model can pick up a specialized workflow without fine-tuning, while a team can revise that workflow independently of the model. The same flexibility turns skills into a software supply-chain surface.
A static checker can spot a suspicious command, an undeclared network call, or instructions that request access outside the stated task. Behavior assembled from several individually plausible steps is harder to catch. A skill might tell an agent to retrieve an installation fragment, save it, and execute it later. The danger is in the composition, not necessarily in any one sentence.
TRUSS treats the complete package as the candidate, not just its main Markdown file. It freezes the package, assigns a content digest, and binds findings to the exact version that produced them. This detail matters in practice. A clean report for version A says nothing about version B after someone changes a helper script.
Static review still has an important job. TRUSS checks functional claims against source and domain evidence, then evaluates the artifact against nine predefined safety properties: control integrity, access boundary, execution integrity, lifecycle isolation, resource boundedness, evidence integrity, authority integrity, composition integrity, and transaction safety.
Some properties can be settled before anything runs. The rest become explicit obligations for the runtime test.
Why the shadow run matters
A candidate that clears the static gate is loaded into a Controllable Execution Environment. The shadow agent receives synthetic task data in a disposable workspace. The skill is mounted read-only, host resources are kept out of the execution namespace, command networking is disabled, and HTTP requests terminate at brokered mock services.
Every requested action passes through an intermediate breaker. Allowed actions reach the sandbox. Rejected actions return a structured blocked result, and that result remains in the trace. A blocked exfiltration attempt is still evidence that the skill induced unsafe behavior. Containing the damage should not turn the underlying request into a pass.
The trace joins tool calls, observations, broker decisions, causal parents, final state, and the candidate digest. TRUSS checks whether the task succeeded and whether the trajectory respected the relevant safety properties. It links failures back to the responsible skill content, revises that content, and sends the new package through the full static and dynamic process again.
A final answer cannot expose this whole path. An agent may finish the requested task after reading an unauthorized file or making an unintended network request. Asking another model whether the skill looks safe misses plenty too. On 168 blinded SkillInject artifacts, a direct LLM checker reached 44.64% precision and 19.05% recall. Static inspection reached 81.55% precision and 94.05% recall. The complete TRUSS setup reported 100.00% precision, recall, and F1 on the matched benchmark.
The perfect score belongs to those 84 clean and 84 injected artifacts. It is not a universal detection guarantee.
Repair helped, but the executor still mattered
TRUSS also repaired 155 cases from SkillSafetyBench. With GPT 5.5, attack success fell from 38.71% to 19.35%. With GPT 5.4, it fell from 46.45% to 29.68%. Neither configuration introduced an attack in a previously safe case.
The remaining failures are as useful as the headline numbers. Safe task completion after repair was 52.90% with GPT 5.5 and 23.87% with GPT 5.4. The same process retained very different utility with different target models. Any certification record is therefore conditional on the executor, harness, tools, scenarios, and policies that produced it.
The 100.00% Security result on SkillGenBench needs the same qualification. Security in this experiment is the mean pass rate over three controlled property probes for each available method cell. It does not show that all nine properties were exhaustively tested under every future prompt and environment. The authors name scenario coverage and executor behavior as remaining limitations.
A practical acceptance process
TRUSS is a research framework, not a new badge that makes deployment review unnecessary. Its design does offer a useful acceptance process for teams using skills with coding agents:
- Version the whole package. Instructions, scripts, templates, and referenced resources belong under one immutable digest.
- Define authority before testing. Record allowed files, tools, domains, transactions, persistence, and user approvals.
- Keep the evidence separate. Static checks cover declared capabilities and suspicious content. Controlled execution covers state-dependent actions and multi-step composition.
- Compare with no skill. Require a net task improvement instead of crediting work the base agent already handled.
- Score safe completion. Success requires the intended result and an execution that stays within authority.
- Run every revision again. Removing an unsafe path can break the useful workflow. Restoring utility can open another risky capability.
This approach complements instruction linting tools such as katalint, but goes beyond the words in the file. It also clarifies the trust boundary behind the MCP draft for discoverable skills: discovery makes a procedure available, not trustworthy. And it adds a behavioral layer to the cost and containment questions measured by Boundary-Bench.
The useful lesson is broader than TRUSS itself. Treat a skill as code plus delegated authority. Before release, a team should be able to name the exact package version, show its permitted reach, and inspect every action it attempted in a disposable environment. Without that record, the team has reviewed a document. It has not tested the agent behavior the document creates.
