The typical request is mostly old context

The public AgentX v1.0 dataset contains 393 sessions selected from a much larger internal proxy corpus. In that subset, the median input length is 142,000 tokens. The median output is only 444 tokens.

Recomputing the entire input on every turn wastes work because most of it is the same conversation prefix sent earlier. A serving system wants to preserve the corresponding key-value cache, route the next request to wherever that cache lives, and avoid moving more state than necessary.

The benchmark includes contexts up to one million tokens. A separate 256k version supports models with shorter limits. Larger windows let a model see more conversation, but they also make cache capacity an operational constraint. When high-bandwidth memory fills, a server may offload cache state to CPU memory and restore it later. The agent experiences that transfer as latency, so the benchmark needs to include it.

Tool use creates pauses, then bursts

AgentX preserves time between turns. The median recorded inter-turn delay is 3.84 seconds, representing local tool work or other client-side activity. About 10% of delays exceed one minute, which the authors say likely includes waiting for a human.

A pause creates a cache-retention decision. Evict a session too quickly and its next turn pays a large prefill cost. Keep every dormant session resident and active work can exhaust memory.

Subagents add another pattern. In the public subset, 175 sessions, about 44%, contain at least one subagent. The dataset records 1,697 subagent rollouts, with a median of four in sessions that use them. Median subagent wall time is 2.27 minutes.

The trace replayer models each session as a directed acyclic graph. Requests wait on recorded delays and predecessor completion, so a branch can create a burst and the main agent can wait for several branches to join. A constant requests-per-second generator misses that coordination.

Tokens per second is not enough

Aggregate throughput still matters. It does not say whether an interactive agent feels responsive. AgentX records time to first token, end-to-end latency, inter-token latency, throughput, cache behavior, and per-user interactivity across concurrency sweeps. Its results pipeline retains request accounting and connects normalized results to raw replay sidecars and workflow provenance.

The tradeoff appears at high load. Batching can raise total tokens per second while delaying the first token for each user. Cache offload can support more long sessions while adding transfer time. A router can balance GPUs evenly while destroying prefix locality. These systems can exchange one kind of performance for another.

For an internal benchmark, set an acceptable first-token limit and per-user generation rate before selecting the highest-throughput point. A background refactoring agent can tolerate a different latency profile from an agent that stops after every tool call and waits for a developer.

Privacy requires an approximate replay

SemiAnalysis says its collection proxy recorded more than 8,000 sessions, 3.4 million requests, and 610 billion tokens. The public dataset is a representative subset, not the raw conversations.

It contains no original prompts, source code, tool arguments, or tool results. The pipeline tokenizes content, groups it into 64-token blocks, and replaces each block with a session-scoped chained hash. Matching prefixes remain matching prefixes; replacement tokens supply replay content without revealing the original text.

This technique creates uncertainty. The collector cannot see provider-side chat templates, proprietary tokenizers, hidden server context, or the full token treatment of images and documents. Encrypted reasoning is opaque. AgentX uses deterministic placeholders and model-specific padding to estimate the lengths seen by the original server.

The authors also note that workload distributions depend on the harness and tokenizer. A minimalist agent will not resemble a context-heavy one. The public subset reflects the team's traffic, not every coding team. AgentX measures inference serving under a reconstructed workload; it does not measure whether a model writes correct code.

Reproduce your own traffic shape

AgentX's open trace-replay harness works best as a template rather than a universal answer. A team can record provider-independent metadata without storing source text:

• input and output token counts per turn;
• stable session and branch identifiers;
• prefix overlap or cache-hit evidence;
• tool delays and human pauses;
• parent, spawn, and join timing for subagents;
• first-token, end-to-end, and per-user generation latency.

Teams routing several models through api.ish.chat can keep this replay schema consistent across providers, then compare serving choices against their own session mix. The same separation applies on the client side: AgentPulse tracks whether a coding session is converging or stuck, while AgentX asks whether the inference system can serve that session efficiently.

An agent benchmark should preserve the dependencies that make an agent a workload: old context, local pauses, cache location, and parallel branches. Remove them, and a fixed-sequence result says little about the system behavior a long coding session will encounter.