Agent context budgets: when to retrieve, cache, trim, and compact
An AI agent can fill its context window before it finishes the job. System instructions and tool schemas take the first share. User messages, file contents, search results, and tool output pile on with each turn. A larger window buys time, but it cannot decide which details are still worth the model's attention.
That decision needs an explicit budget. Reusable instructions, working state, raw evidence, and completed work have different lifetimes. They should not all remain in the prompt until the API rejects the next request.
Retrieval, prompt caching, trimming, compaction, and persistent memory each handle a different part of the load. A reliable agent uses them deliberately instead of treating "more context" as the default.
Measure what fills the window
Record input tokens by category at the beginning and end of representative runs:
| Context category | Typical examples | Useful question |
|---|---|---|
| Fixed instructions | system prompt, policy, output contract | Does this need to appear on every turn? |
| Tool definitions | function schemas, MCP tools | Does this tool need to be loaded before it is selected? |
| Working state | current plan, open decisions, recent messages | Would removing this change the next action? |
| Evidence | files, logs, search results, database rows | Can the agent fetch it again by ID or query? |
| Completed work | old tool results, resolved steps | Is a concise outcome enough now? |
The categories point to separate causes. A large fixed prefix costs money repeatedly. An oversized tool catalog consumes tokens before work begins. Raw evidence expands during the run, while completed steps remain long after their details stop helping.
Anthropic's context engineering guidance recommends finding the smallest set of high-signal tokens that supports the desired behavior. That is a more useful target than sending as much text as the window allows.
Retrieve facts instead of carrying them
Durable source material can live outside the conversation. The working state needs a file path, document ID, query, or artifact URL, plus enough context to know when to fetch the source again.
A coding agent might keep the target file, the relevant test failure, and the current patch in view. It can search for other files when they become relevant. A support agent needs the ticket and the applicable policy excerpt, while customer history can remain behind a scoped lookup tool.
Retrieval can fail if a reference no longer resolves or returns different data. Use immutable artifact IDs or versions when exact evidence matters. For a mutable source, record when it was read and which fields affected the decision.
Persistent memory fits in this external layer. Anthropic's memory tool documentation describes a client-side store where the model requests file operations and the application controls the underlying storage. A later conversation can retrieve saved project facts without loading every previous session.
Memory should not become a transcript archive. Stable preferences, decisions, identifiers, and lessons may change future work. Secrets, unverified guesses, and routine intermediate responses do not belong there.
Cache stable prefixes without confusing caching and cleanup
Prompt caching lowers the cost of processing repeated prefixes. It does not shrink the number of tokens in the context window. That makes it useful for a policy block or tool set that appears on many turns, but it will not rescue a conversation already packed with stale results.
Cacheable material usually works best when it stays stable and appears early: tool definitions, system instructions, canonical examples, then changing messages and tool output. Provider rules differ, but changes such as reordering tools or editing a shared instruction can invalidate the reusable prefix.
Anthropic's tool context guide treats prompt caching, tool search, and context editing as separate mechanisms. Tool search avoids loading unused definitions. Caching lowers repeated processing cost. Context editing removes old results. They can work together because each acts on a different source of context pressure.
Cache reads and writes need measurement. Enabling a cache around a prefix that changes on every request does not make that prefix stable.
Trim raw tool results once the outcome is known
Search responses, compiler logs, and file listings often dominate an agent run. One may be essential for a decision, then become useless two turns later.
After a milestone, a bulky result can become a structured record:
{
"step": "checkout tests",
"status": "failed",
"evidence": ["artifact:test-run-1842"],
"finding": "tax rounding differs for JPY orders",
"next": "inspect money.ts and the JPY fixture"
}
This record retains the outcome, a handle for the evidence, and the next action. The full log remains retrievable. Raw data should stay in context when the next step must quote it, compare individual rows, or verify an exact byte sequence.
Provider-managed editing can automate part of the cleanup. Anthropic's context editing documentation can clear older tool results after a configured threshold is crossed. The service inserts placeholders for cleared results, while the client keeps its unmodified conversation history. Clearing also interacts with prompt caching. Removing a meaningful block at once is preferable to repeatedly invalidating a cached prefix for small savings.
Compact at milestones
Compaction summarizes or encodes what the agent needs to continue while removing much of the earlier token load. It fits the boundary between phases, when decisions, unresolved blockers, identifiers, and the next goal still matter but the full working history does not.
OpenAI exposes POST /responses/compact for Responses API conversations. The API reference says the result contains user messages followed by a compaction item, along with token usage for the compaction pass. That item is continuation state. Application code should not parse it as a business record.
Useful boundaries include the end of research, completion of a patch, or the point before a deployment review. Running compaction after every turn adds work and may repeatedly compress details that the next step still needs in full.
Structured application state remains necessary alongside provider compaction. An opaque continuation item should not be the only record of a changed database row, an outstanding approval, or an artifact that still needs delivery.
Adopt a small operating policy
One workable policy has five rules:
- Load only the tools and evidence needed for the current phase.
- Cache stable prefixes and measure actual reuse.
- Store large evidence externally with durable references.
- Replace completed tool output with a short outcome record.
- Compact after a meaningful milestone or before the context limit becomes urgent.
Evaluate it on real tasks. Track completion, factual errors, tool retries, input tokens, cache reads, and compaction frequency. Check whether the agent can explain its last consequential action from durable records after the original tool output is gone.
Permission boundaries apply to retrieval and memory too. Saved state can expose data that the current user should not see, so identity and resource scope must be checked before the tool returns it. The least-privilege checklist for AI agent tools covers that layer.
Model comparisons also need a consistent context policy. Giving one model a clean working set and another a swollen transcript measures orchestration as much as model behavior. ISH chat can compare how models interpret the same working set, while the ISH API dashboard keeps usage visible during longer API tests.
A context policy is working when the agent can finish a long task, recover the evidence behind its decisions, and continue after compaction without inventing missing state.
