COBRA-Skills cut reported optimization cost 55% to 58% versus SkillOpt
Writing another SKILL.md is easy. Finding out whether it changes an agent's behavior across enough tasks to deserve trust is the expensive part.
A new preprint, COBRA-Skills, treats that work as a budget-allocation problem. It keeps a population of candidate skills, predicts which ones are worth testing, and reserves some evaluation capacity for candidates whose value is uncertain. Across six benchmarks and three target models, the authors report the highest average score among the methods they compared. Their estimated optimization cost was 55% to 58% lower than SkillOpt's.
The number needs context. Cost here is calculated from tokens using a fixed price table, not measured as elapsed time or infrastructure cost. This is a v1 preprint, and the method still used roughly 110 million to 125 million target-model tokens per model across the full benchmark suite. COBRA-Skills does not make skill optimization cheap. It offers evidence that testing every candidate with equal enthusiasm is wasteful.
Selection and revision are different jobs
COBRA-Skills starts with ten candidate skills. It embeds each skill and treats it as an arm in a contextual bandit. A small neural predictor estimates likely reward. A LinearUCB bonus lifts candidates that have not been explored enough, keeping the search from collapsing immediately around an early favorite.
The selected skill is run by the target agent. Its reward and execution trajectories enter the optimization history. Crucially, the population is not rewritten after every result. At scheduled updates, three operators create replacements:
- Regeneration produces an independent skill from no-skill trajectories.
- Rollout mutation makes a conservative edit using successes and failures from an evaluated skill.
- Crossover retains a strong backbone, borrows compatible strategies from another strong candidate, and treats weak candidates as negative evidence.
Low-priority skills leave the pool. New candidates receive no inherited reward and must earn a score through execution.
Selection asks which candidate deserves the next expensive test. Evolution asks what should enter the pool next. A typical homegrown loop asks an LLM to revise a prompt after every failure, then evaluates whatever appeared. COBRA-Skills puts those decisions on separate schedules.
What the comparison actually covers
The six benchmarks are SearchQA, SpreadsheetBench, DocVQA, LiveMath, SocialMaze, and ALFWorld. Together they cover question answering, spreadsheets, document vision, mathematics, social deduction, and interactive household tasks. All methods used a held-out test set of 100 examples per benchmark. COBRA-Skills optimized on 50 unique examples per benchmark over 30 rounds with a population of ten.
The target models were Qwen3.6-35B-A3B, GPT-5.4-Nano, and Gemma-4-26B-A4B-it. Compared with no-skill agents, COBRA-Skills raised their average scores by 13.1, 26.9, and 22.5 percentage points. Compared with SkillOpt, the gains were 3.9, 3.2, and 1.3 points.
The cost table is more useful than its headline percentage. COBRA-Skills cost $54.10, $58.46, and $38.98 under the authors' accounting, versus $121.02, $133.59, and $92.14 for SkillOpt. Those totals come from recorded input and output tokens multiplied by fixed per-million-token prices. The paper notes that provider prices change.
COBRA-Skills used 67% to 80% fewer teaching-model tokens than SkillOpt, but did not always use fewer target-model tokens. On Qwen, it used about 116 million target tokens versus SkillOpt's 94 million. On GPT-5.4-Nano, it used 110 million versus 169 million. On Gemma, it used 125 million versus 89 million. Scheduled skill rewriting reduced expensive teacher calls; it did not erase the need for target-agent evaluation.
If your teacher is local and your target model is the expensive component, your cost curve may not resemble the paper's.
Stress tests and remaining gaps
An ablation compared the full system with random candidate selection, a fixed pool, and best-of-30 selection. Removing the bandit lowered the average Qwen result by 2.2 points. Removing evolution lowered it by 2.4. Best-of-30 trailed by 2.5. Both adaptive selection and population refresh contributed within this setup.
The authors moved the optimized skills into Codex and Claude Code harnesses while keeping Qwen as the target model. COBRA-Skills had the highest average score in both tables, although its lead over SkillOpt under Codex was 1.3 points and results varied by benchmark. In a separate transfer test, 34 of 36 skills improved on the receiving model's no-skill baseline.
No independent team has replicated these results yet. Several benchmark cells have large standard errors. The authors also excluded OfficeQA because they could not reproduce SkillOpt's released configuration closely enough for a fair comparison. Omitting that benchmark was reasonable, but the episode shows how a shaky baseline can limit the certainty of an entire comparison.
Borrow the discipline before the machinery
The Apache-2.0 repository includes benchmark adapters, fixed split manifests, tests, resumable artifacts, and native, Codex, and Claude Code harness options. A smaller team can start with a much lighter process:
- Maintain several explicit skill variants instead of one sacred prompt.
- Test them on real tasks. An LLM's review of the prose is not an outcome score.
- Track uncertainty beside the mean, leaving some capacity for an under-tested candidate.
- Revise only after enough executions reveal a recurring success or failure.
- Reset the score when a revision enters the pool.
- Hold back a test set, then report tokens, elapsed time, and money as separate measurements.
Simple baselines still matter. Compare no skill, one evidence-grounded skill, and a small fixed pool before adding bandit machinery. SkillsBench reports that skill effects vary across tasks and models, while SkillOpt represents the iterative approach used in COBRA-Skills' main comparison.
Execution records must also reflect what the agent actually saw. A coding assistant that updates its edit history after a formatter changes the file has better evidence for its next action than one reasoning from a stale patch. And subagent experiments that reduced peak context while increasing total tokens show why a single efficiency number rarely tells the whole story.
A skill file is a behavioral intervention with an uncertain return, not static documentation. Once evaluations are treated as scarce decisions, endlessly polishing instructions before running them looks like the wrong loop.
