title: "CCBench gave coding agents 20 minutes. One timed out on about a quarter of tasks" slug: "ccbench-coding-agent-20-minute-timeout" excerpt: "CCBench tests agents on messy private code, and its timeout result shows why speed belongs in the benchmark rather than a footnote." category: "Developer Tools" tags:
- "CCBench"
- "coding agents"
- "benchmarks"
- "timeouts"
- "CodeCrafters" author: "ISH Team" reading_time: 7
CCBench gives each coding agent 20 minutes to add a real feature to a small, human-written codebase. Gemini CLI with Gemini 3 Pro Preview exceeded that limit on about 25% of its tasks.
The note sits beneath the leaderboard, yet it changes how the 47.6% success rate reads. Some runs did not end with a wrong patch. They simply did not finish within the operating budget.
For a team choosing a coding agent, elapsed time is part of the product. Solving a task after an hour is different from reliably returning a reviewable patch before a developer's next meeting. CCBench shows why the benchmark clock belongs beside the pass rate.
What CCBench puts under the clock
CCBench comes from CodeCrafters, the company behind build-your-own programming challenges. Its repository contains 187 tasks, with the published leaderboard based on roughly 180. Each task starts from a private user submission under 10,000 lines of code, then asks the agent to implement another stage of the challenge.
The requests are recognizable engineering work: add authentication to a Redis implementation, backreferences to a regular-expression engine, autocomplete to a shell, functions to an interpreter, or partition-listing APIs to a Kafka implementation. Agents receive the codebase, a detailed feature instruction, and information about how tests will run.
The example backreferences task reads more like a staged product requirement than a one-line bug report. The agent must preserve earlier behavior while adding capture groups, numeric backreferences, validation, and matching rules.
Official CodeCrafters test runners determine success. Each task also has a reference solution that can run through Harbor's oracle agent. The task files set a 1,200-second agent timeout, plus separate limits for tests and environment building.
Private code changes the contamination question
Popular coding benchmarks often draw issues from public open-source repositories. Those tasks are realistic, but the repository, issue, or fix may have appeared in training data. CCBench uses private submissions made by people working through the CodeCrafters catalog. The starting implementations are therefore less likely to be memorized verbatim.
That claim needs a boundary. The benchmark is public now, so later models may encounter its tasks. The projects also reimplement familiar systems such as Git, Redis, SQLite, Kafka, and shells. A model may know those protocols and common implementation patterns even when it has never seen the submitted code.
The safer claim is that CCBench reduces direct exposure to the exact starting repository at the time the tasks were assembled. Prior knowledge remains, and the benchmark cannot prove that every model lacks related examples.
The code is messy by design. User submissions vary in language, architecture, naming, and completeness. That makes the benchmark closer to the small private repositories many developers maintain. It also makes the task sample different from mature open-source projects with established conventions and extensive tests.
The clock changes what the score means
The February 11, 2026 leaderboard puts Codex CLI with GPT-5.2-codex first at 75.4%, followed by Claude Code with Opus 4.6 at 72.7%. Claude Code with Opus 4.5 scores 58.3%, Gemini CLI with Gemini 3 Flash Preview 51.3%, and Gemini CLI with Gemini 3 Pro Preview 47.6%. Four other configurations range from 42.2% to 21.9%.
Each row measures a system rather than an isolated model. The result combines a model with its coding agent, prompts, tools, defaults, and stopping behavior. Comparing GPT-5.2-codex in Codex CLI with Opus 4.6 in Claude Code does not reveal what would happen if both models used the same agent loop.
The Gemini 3 Pro timeout note adds another confound and a useful signal. A fixed wall-clock budget rewards systems that inspect, edit, test, and stop efficiently. It penalizes long reasoning, slow tool sequences, repeated exploration, and infrastructure latency together. Without the trajectory, the score cannot show which factor caused a particular timeout.
CCBench publishes run logs for inspection and uses the open Harbor task format. A more informative leaderboard would break outcomes into passed, test-failed, agent-error, and timed-out groups for every configuration. Median and tail latency among completed tasks belong there too. One timeout percentage exposes the issue; a full latency table would let readers compare it.
Reproduce the clock that matters to your team
A local evaluation benefits from two limits. The shorter one should match the developer-facing service level, perhaps ten or twenty minutes for an interactive task. A longer diagnostic ceiling can reveal whether the agent eventually succeeds. Together they separate "cannot solve" from "did not solve soon enough."
Timeout policy must stay identical across models, and each task needs repeated runs. Cold container starts, dependency downloads, and provider latency can turn one attempt into noise. Include environment setup in the main clock when users experience that wait. When setup exists only for the benchmark, time it separately.
An expired run should keep its trajectory and partial artifacts. The last test result, modified files, tool sequence, and reason for stopping can distinguish a nearly finished patch from an agent looping on the wrong approach. Our guide to grading the full coding-agent trajectory covers that evidence, while CAR-bench's repeated-run analysis explains why one successful attempt is not reliability.
For comparisons behind api.ish.chat, pin exact model IDs, prompts, tool permissions, retry rules, and repository snapshots. The shared endpoint removes integration differences, but it does not remove evaluation choices. Those choices need to travel with the score.
What this leaderboard does not establish
CCBench is maintained by the company that owns the source platform and tests. Its tasks come from people who chose build-your-own challenges, not a random sample of company repositories. The public table reports one aggregate success rate per agent configuration without confidence intervals, per-language splits, token costs, or complete latency distributions. The latest listed results are dated February 11, 2026 even though the repository has received later task fixes.
The benchmark still covers a part of software work that polished public repositories miss: private, imperfect starting code. Its 20-minute rule asks a question many accuracy tables leave out. Did the agent produce a correct patch while the answer was still useful?
