WebDev-Skills-Bench: matched skills cut Pass@2 and raised token use
Agent Skills package framework rules, anti-patterns, examples, and helper files for reuse by coding agents. Activation usually begins with a relevance match. A new web-development benchmark shows why that match should be treated as a first filter rather than a verdict.
WebDev-Skills-Bench, released as an August 24 preprint by Ziyue Yang and Fan Ding, tested 31 public WebDev Skills on 50 projects and 1,000 ordered tasks. The researchers routed each Skill only to projects whose stack it directly covered. Even with that conservative matching, target Skill injection lowered mean Pass@2 on all four tested models by 1.3 to 4.2 percentage points and increased token use by 72% to 394%.
Some Skill-project pairs improved on every model, so the study does not support removing Skills wholesale. It does show that relevance metadata cannot stand in for an evaluation.
A same-length control separates two problems
The Agent Skills specification defines a Skill around a required SKILL.md file, with optional scripts, references, and assets. Clients load metadata first, then load the full instruction file when they activate the Skill. Longer supporting material can stay in separate files and be read only when needed.
WebDev-Skills-Bench mirrors that design. It injects only SKILL.md into the prompt and mounts auxiliary files in the workspace. The benchmark then compares four conditions:
- C0 uses no Skill.
- C1 injects the matched target Skill.
- C2 injects an irrelevant Skill within 5% of the target's byte length.
- C3 removes selected parts of a helpful Skill to test positive rules, anti-patterns, and example code separately.
C2 lets the researchers separate two possible causes of a decline: the target instructions and the addition of more prompt text. A benchmark without the same-length control would mix those effects.
The task base is ByteDance's open-source Web-Bench, which contains 50 projects with 20 sequential tasks each. The projects cover 11 stack categories, including React, Vue, Angular, Svelte, back-end frameworks, databases, CSS, Canvas, SVG, Three.js, bundlers, and DOM applications. Verification uses deterministic Playwright tests rather than an LLM judge.
Two authors independently reviewed all 1,550 possible Skill-project pairs. They agreed on 1,495 and resolved the remaining 55 by discussion, yielding 117 "core" pairs across all 50 projects. That routing work matters because a benchmark that attaches irrelevant framework advice would mostly measure obvious mismatches.
Average performance fell on all four models
The paper evaluated Claude Sonnet 4, GPT-5.1, Qwen3-Coder-30B-A3B, and DeepSeek-V4-flash. Each model-condition-pair cell received three runs with greedy decoding, a 64k maximum token budget, and a clean workspace before every execution.
Target Skill injection changed mean Pass@2 as follows:
| Model | Pass@2 change | Token change | Pairs that improved |
|---|---|---|---|
| Claude Sonnet 4 | -4.2 pp | +72% | 30% |
| GPT-5.1 | -1.3 pp | +74% | 35% |
| Qwen3 Coder 30B | -2.3 pp | +91% | 17% |
| DeepSeek V4 Flash | -2.0 pp | +394% | 36% |
Three Pass@2 confidence intervals excluded zero. GPT-5.1's Pass@2 interval marginally included zero, although its Pass@1 decline did not. The DeepSeek token percentage also needs care: the authors say early failures in the no-Skill condition reduced its denominator and amplified the relative increase.
Skills still helped a minority of pairings. A deployment needs to identify that minority before every request pays the context cost.
Easy tasks took the clearest hit
Skill injection produced its largest reliable losses on easy tasks. Across the four models, the average easy-task Pass@2 decline ranged from 4.0 points for GPT-5.1 to 10.7 points for Qwen, with every confidence interval excluding zero. Moderate and challenging buckets were smaller and noisier, and did not show one consistent direction.
The authors describe a retry lock-in mechanism. Web-Bench allows two attempts. On an early task, a model may recover by changing a button label, class name, or element structure during the second attempt. A Skill can anchor the model to one convention, turning a cheap first error into the same error twice.
Aggregate scores hide this failure. A Skill may look sensible against the whole repository and still reduce flexibility on routine work. Results should be split by task type and chain position, not only final project completion.
Length distraction and content misalignment are different failures
The irrelevant C2 control split the models into two patterns. For Sonnet and Qwen, an equally long irrelevant Skill reproduced most of the loss, which the paper calls length distraction. GPT-5.1 and DeepSeek were less affected by added length, but target Skill content still reduced Pass@2 by 1.1 and 1.4 points relative to the length control. The paper calls that content misalignment.
Length distraction calls for shorter activation instructions and progressive disclosure. Content misalignment calls for tighter routing, version checks, and removal of rules that conflict with the project. Our guide to agent context budgets covers the cost side; repository instruction design covers how to keep guidance concrete and scoped.
Cross-model transfer was weak. Pair-level Skill effects had absolute Pearson correlations no higher than 0.12 across models. A Skill leaderboard built with one model therefore says little about another. Static labels such as "recommended for React" leave out the model and the task distribution.
Anti-patterns beat example-heavy guidance
In its focused component ablation, the benchmark found that anti-pattern rules carried the most reliable benefit inside helpful Skills. Example code helped weaker models but hurt the strongest model in the study. This part of the experiment covers selected helpful pairs rather than the complete benchmark, so it should guide another test, not become a universal writing rule.
A practical Skill audit can start with four runs:
- Run the task without the Skill.
- Run it with the target Skill.
- Replace the target with irrelevant text of similar length.
- Test a shorter version that keeps prohibitions and removes most examples.
Keep the model, task order, workspace reset, decoding settings, and verifier fixed. Track correctness, completion depth, retries, and total tokens. A Skill earns default activation only when it beats the no-Skill baseline often enough to justify its recurring cost.
What the study cannot settle
This is a first-version preprint covering web development, 31 third-party Skills, four models, and the Web-Bench harness. Deterministic tests improve repeatability, but 50 benchmark projects are not every production codebase. The authors also made the routing judgments themselves. Other models, agent harnesses, Skill activation policies, or narrower task distributions may behave differently.
Other benchmarks point in different directions. SkillsBench reported a 16.2-point average gain from curated Skills across diverse domains, while SWE-Skills-Bench found that 39 of 49 software-engineering Skills produced no pass-rate improvement. WebDev-Skills-Bench offers one explanation for the difference: Skill value belongs to a Skill-project-model combination, not to the file alone.
A successful relevance match is not proof of benefit. A registry can ask a cheaper, stricter question: compared with no Skill and the same amount of irrelevant text, did this package improve the work?
