Instrumenting AI agents: the traces that make failures explainable
An agent can produce a bad answer after a successful model call. It may retrieve the wrong document, choose an unsuitable tool, retry a request until it times out, or receive a tool response that changes its plan. A single application log line such as agent failed does not tell you which of those happened.
Treat one user-visible task as one trace. Within that trace, record the model calls, retrieval work, tool calls, handoffs, and final outcome. That gives an engineer a sequence to inspect instead of a pile of unrelated logs.
Recording every prompt and tool response by default creates privacy and storage problems. Those fields can contain user data, secrets, or more text than a telemetry system can handle comfortably. Use a small, structured trace, and keep content capture separate and tightly controlled.
Start with a request-level trace
Create a trace when the application accepts a task, not when it first calls a model. Give it a stable ID that also appears in your application logs, job queue records, and HTTP responses. If your model provider returns a request ID, log that too. OpenAI recommends logging request IDs in production so support can investigate a specific API request, and its SDKs expose that value on top-level response objects.
Use low-cardinality attributes for the trace header:
- workflow name and deployed version
- tenant or project identifier, if your privacy model permits it
- environment and region
- selected model and provider
- outcome class: completed, blocked, failed, or cancelled
Do not put a raw user ID, prompt text, document body, access token, or tool arguments in the trace header. A trace is usually easy to search. That convenience makes it a poor default home for sensitive or high-cardinality data.
OpenTelemetry's GenAI conventions include a gen_ai.workflow.name attribute and well-known operation names such as invoke_agent, chat, retrieval, and execute_tool. The conventions are still under development, so use them where they fit and keep your own names consistent rather than treating every field as permanent.
Make the agent's decisions visible as spans
The trace should match the work the agent did. A practical tree looks like this:
invoke_agent: support-answer
├─ retrieval: search-help-center
├─ chat: decide-next-step
├─ execute_tool: get-account-status
├─ chat: write-answer
└─ emit_response
Attach duration, status, retry count, and a compact reason to each span. For a model span, record the provider's model identifier and input/output token totals. For retrieval, record the index or corpus name, result count, and a retrieval version. For a tool span, record the tool name, operation type, target service, HTTP status or error class, and a sanitized argument shape.
get_account_status with { "account_id": "[redacted]" } is often enough to diagnose a bad tool selection. It is safer than exporting the full request. If arguments are needed for an incident, store an encrypted, access-controlled event record elsewhere and link it with an opaque event ID.
The OpenTelemetry registry defines attributes for tool call IDs, arguments, and results, but also says large optional properties should not be populated by default. Capture structured metadata everywhere; enable content capture only for a short-lived, reviewed debugging path.
Separate failure from a disappointing answer
An HTTP 200 means the API returned a response. It does not prove that the agent completed the task.
Define a small outcome vocabulary before you build dashboards. For example:
completed: the agent produced a response and met the application's completion rule.blocked: it needed approval, a missing credential, or more user input.failed_tool: a required tool call could not complete.failed_policy: a guardrail or permission check stopped the action.failed_budget: the run reached a token, time, or tool-call limit.cancelled: the user or caller stopped the run.
Emit the final outcome on the root span and include the same value in a structured application event. When a run fails, add the first failing span ID and the final decision reason. This turns a support question such as "why did it stop?" into a trace lookup instead of a reconstruction project.
For retries, record each attempt as a child span. Include whether the retry was automatic, the error class, and the delay chosen by the caller. Do not collapse five retries into one duration. A trace that shows one slow tool span and four fast rate-limit errors points to a different fix than a trace with five slow upstream requests.
Measure the things that change operating cost
Traces explain an individual run. Metrics show whether a change helped or hurt across many runs.
Start with four views:
- end-to-end duration, broken down by workflow and outcome;
- tool error rate and retry rate, broken down by tool name;
- input, output, and reasoning-token usage, broken down by model and workflow; and
- completion, blocked, and failure counts, broken down by deployed version.
OpenTelemetry's current GenAI material describes separate token-usage and operation-duration metrics. Pair those with your own outcome counter. Token use alone does not explain whether a costly workflow succeeds, and completion rate alone can hide a model or prompt regression that doubled cost.
Avoid a dashboard that groups every failure under "agent error." A tool's 429 responses, malformed tool arguments, policy denials, and model refusals belong in different slices. The point is to decide what to change: a rate limit, a schema, a permission grant, or the workflow itself.
Build redaction into the instrumentation boundary
Telemetry has a habit of spreading. It moves through collectors, vendors, alert payloads, and incident exports. Redact before export, not after someone finds a secret in a dashboard.
Use an allowlist for attributes. Hash or tokenize identifiers when correlation is necessary. Strip authorization headers, cookies, secrets, document text, and free-form user messages unless an explicit incident workflow allows them. Set retention rules for detailed events, and make content capture auditable.
Tool results deserve the same treatment as prompts. A database lookup or ticketing API can return personal data even when the model request did not contain it. The least-privilege rules you apply to agent tools should extend to their telemetry. A least-privilege checklist for AI agent tools is a useful companion when you are deciding which tools can run without approval.
Ship a narrow first version
Instrument one workflow that already causes debugging pain. Add the root trace, model and tool spans, a final outcome, token totals, and redaction tests. Then run a handful of known scenarios: a normal completion, a tool timeout, a rate limit, a policy block, and a user cancellation.
Look at the trace for each scenario without reading the application code. If it does not answer what the agent tried, where time went, and why the run ended, add one field or span at a time. Resist the urge to turn every internal thought into telemetry. Agents already have a context budget to manage; your observability system needs one too. Agent context budgets covers the related problem of keeping long-running workflows focused.
