Boundary-Bench finds sandbox hardening can raise agent cost by 167.3%
Security controls do not impose one predictable tax on coding agents. In Boundary-Bench's current Terminal-Bench 2.1 results, the strictest policy reduced success rates by 7.1 to 18.3 percentage points across the listed agent and model combinations. Cost moved differently: one combination rose 16.0%, while another rose 167.3%.
Selecting an agent in an unrestricted evaluation and adding a hardened runtime later may change both the ranking and the budget. Boundary-Bench measures that change directly.
Boundary-Bench is an open-source benchmark from Dotan Davidovich, Yair Amar, Hai Rozencwajg, and Or Hiltch. It runs coding-agent harnesses on Terminal-Bench tasks inside Daytona sandboxes, routes models through OpenRouter, and records Inspect AI .eval logs. The accompanying paper, submitted August 2, 2026, describes the problem as policy-graded evaluation: capability is measured while the environment progressively removes privileges.
What the strict policy changes
The published high-nist level combines three restrictions. Network access uses a per-task allowlist. The operating system and home directory are read-only while the workspace stays writable. The agent runs as a non-admin user with no_new_privs and dropped capabilities.
These are native Linux controls, including nftables, read-only bind remounts, setpriv, and Landlock. A blocked action appears to the agent as an ordinary operating-system error such as EROFS, EPERM, or a refused connection. There is no friendly benchmark shim explaining the intended workaround.
The control arm allows full egress, an open filesystem, and root. Boundary-Bench also exposes individual options for non-root execution, blocking escalation, freezing the home directory, making the operating system read-only, and restricting egress. Teams can test the policy their production environment uses instead of treating "sandboxed" as a single setting.
The model endpoint is exempt from egress restrictions. Inbound traffic is not restricted because the sandbox exposes no services. Those details matter when interpreting the result: the benchmark constrains the work environment without cutting the agent off from its model.
The leaderboard changes when the walls go up
The current Terminal-Bench 2.1 leaderboard covers 89 tasks, three trials per agent-task pair, and high reasoning effort. Its uncertainty values are one standard deviation across those trials.
Grok Build with Grok 4.5 falls from 82.0% success in the control environment to 74.9% under high-nist, a 7.1-point loss. That is the smallest listed success drop. Its cost rises from $46.2 to $123.5, an increase of 167.3%, the largest on the table.
Codex with GPT-5.6 Sol falls from 83.9% to 74.2%, while cost rises 29.2%. Claude Code with Fable 5 drops from 79.0% to 67.8% and costs 18.6% more. At the other end, Codex with GPT-5.6 Luna loses 18.0 points but adds only 16.0% to cost. Claude Code with Sonnet 5 records the largest success loss, 18.3 points, with a 21.4% cost increase.
These are agent, model, harness, policy, and task-bundle results. They are not model scores in isolation. The table also mixes very different control costs, from $13.3 for the Luna bundle to $316.4 for Fable 5. A percentage increase does not replace the absolute bill, and one three-trial mean does not establish a permanent ordering.
Some setups preserve task success by spending much more after a denial. Others contain the cost increase but fail more tasks. A production comparison needs the success and cost columns together.
A denial often becomes a longer failure
The paper reports that hardening does not usually make an agent stop cleanly at the first blocked action. Runs instead drift toward timeouts or wrong solutions, with the mixture varying by model.
An agent may retry a prohibited operation, search for another route, or continue after a partial workaround. Extra tokens and tool calls can accumulate even when the final verifier still fails. A security policy can improve containment while making an unsuccessful run slower and more expensive.
This is an engineering reason to expose denials in telemetry. Record which policy rule blocked the action, what the agent tried next, how long it continued, and whether it reached a verified alternative. A bare "task failed" result hides the difference between a model that adapts and one that burns its budget against the same wall.
The benchmark's analyze command produces a policy-blockage report from the .eval logs. Its enforcement-probe command tests the preset's native boundaries without running an agent. First prove that the control is enforced. Then measure how the agent behaves inside it.
Seven tasks cannot pass under hardening
Boundary-Bench does not quietly score every failure against the model. Seven of the 89 tasks require an action that the hardened policy forbids, such as system-wide installation, writes under /etc or /var, or access to a host outside the allowlist. The project marks these tasks not_applicable and excludes them from hardened success rates.
Five other tasks use adapted verifiers because the stock verifier checks a root-owned path. The corrected verifier is applied under every policy arm so grading remains consistent. The repository publishes the verifier diffs and per-assertion notes.
This bookkeeping avoids confusing a policy's intended refusal with an agent's inability to solve a permitted task. If a real workflow requires a blocked operation, the team must redesign the task, create a narrowly approved capability, or accept that the policy makes it impossible.
Run the policy before buying the ranking
Boundary-Bench requires Python 3.12 or newer plus OpenRouter and Daytona credentials. A single documented run looks like this:
uv sync --extra harbor --extra daytona
uv run boundarybench run \
--task log-summary-date-ranges \
--policy high-nist \
--harness claude-code \
--model anthropic/claude-opus-5
Start with tasks that resemble your own builds and deployment boundaries. Run an enforcement probe, confirm task applicability, and compare the same harness and model under control and hardened policies. Inspect costs and failure modes rather than copying the public winner.
Our AgentArena guide covers how to build repeatable tasks from your own repository. The least-privilege checklist covers capability scoping outside a benchmark. Boundary-Bench connects those concerns: it measures the work an agent can still finish after the permissions become real.
Benchmark the sandbox policy with the model, harness, task, and budget. The cheapest unrestricted setup may not stay cheap when the operating system starts saying no.
