CollabSkill separates human and agent scores, but not for hiring
Most agent leaderboards remove the human from the test. That produces a clean model score for a workflow that many employers do not actually want: the agent working alone.
CollabSkill, a Stanford SALT Lab framework accepted to COLM 2026, evaluates the team instead. It pairs agents with real workers on occupational tasks, then estimates a separate collaboration skill for each side. In August, the lab published its interaction trajectories and rating implementation, so the method is available for inspection and reproduction.
Its agent ranking differs from rankings based on autonomous work. The human ranking needs more restraint. Although CollabSkill assigns latent scores to people, the paper explicitly warns against using them for employment screening or other consequential decisions.
One outcome, two contributors
The study collected more than 1,500 prompts from 386 sessions with 93 workers recruited through Upwork. Participants were based in the United States and completed tasks matched to their occupational backgrounds across 10 O*NET sectors. Five agents were tested: Claude Cowork, Claude Code, Codex, Manus, and Gemini CLI.
Sessions produced open-ended deliverables including spreadsheets, PDFs, slide decks, ZIP archives, and audio files. The researchers used a two-stage automated grading pipeline. One agent generated a rubric, then two agent judges produced a score from 0 to 100.
A simple average credits an agent for every point earned by its human partner. It can also punish an agent paired with someone who has less experience using AI. CollabSkill models the score differently:
team outcome = agent skill + human skill + observation noise
Each agent and person starts with a Gaussian prior. The estimates update as sessions accumulate. The published leaderboard uses mu - 3 * sigma, a conservative score that penalizes entries with greater uncertainty. The open-source collaborative_gym.eval.collabskill module exposes the implementation and reproduces the released ratings.
This is a statistical decomposition, not a record of who wrote each sentence or repaired each formula. The model infers contributions from patterns across pairings. Its answer depends on the additive assumption, the session scores, the observed people and agents, and the automated grader.
The order changes when a person joins the test
Claude Cowork ranked first on CollabSkill, followed by Claude Code, Codex, Manus, and Gemini CLI. Among the three terminal agents, Claude Code ranked above Codex even though Codex led it in the paper's solo evaluation and the autonomous benchmarks used for comparison.
Claude Cowork and Claude Code both used Claude Sonnet 4.6. Their different results suggest that the interaction surface and harness affect collaborative performance alongside the underlying model. They do not establish that desktop interfaces generally beat terminals. The comparison covers specific agents, tasks, workers, and early-2026 model versions.
A procurement test can therefore select the wrong system for the job. An agent may complete solo tasks well yet communicate poorly, take initiative at the wrong time, or make intervention awkward. If production includes a person, the evaluation should include one too.
The same issue complicates model comparisons. A gateway such as api.ish.chat can route the same task to multiple models, but equal API access does not equalize the interface, tool permissions, context, or collaboration policy. Those belong in the test design.
A human score is not an employment score
Practical familiarity with LLMs predicted stronger collaboration skill in the study. After working with agents, participants reported greater trust and comfort with delegation. They also judged agents capable of more autonomy, while their preferred level of autonomy for meaningful work did not significantly change. Workers can revise their view of what an agent can do without giving up control over important work.
Turning the human-side rating into an employee ranking would ignore the study's design. Its 93 participants do not represent the workforce. The paper defines the human score as a proxy for AI literacy within this collaboration setting and cautions against employment screening. A low estimate might reflect unfamiliarity with one interface, an unlucky pairing, task mismatch, accessibility barriers, or grading noise.
The group-level result is still useful for employers. If experience affects outcomes, a rollout needs practice time and workflow training. A stronger model cannot teach employees when to direct, interrupt, verify, or recover from it.
A fairer internal agent trial
Teams choosing an agent can borrow the experimental shape without adopting the Bayesian model.
- Test representative deliverables through the interfaces employees will use. Isolated prompts are not the real unit of work.
- Give each participant more than one agent, and pair each agent with several participants. These crossover pairings keep one unusual team from deciding the result.
- Keep the deliverable and the process evidence. Record interventions, reversals, tool calls, verification, and repair time. The BLOGish guide to agent traces that explain failures covers the logging side.
- Publish session counts and uncertainty. A leaderboard without either invites false precision. CollabSkill's conservative score is one option.
- Audit the grader for the failures that matter in your domain. Automated judges may reward polish over correctness.
- Keep product evaluation separate from worker evaluation. Aggregate patterns can improve training and interface design. Experimental collaboration estimates should not become performance-management scores.
This is also the case for trajectory-based coding-agent evaluations. The artifact matters, but the route reveals whether the agent was controllable and how much hidden repair produced the apparent success.
Read the data with its limits attached
The CollabSkill trajectories dataset is licensed CC BY 4.0. It links interaction logs to the study and can support reproduction, alternate analyses, and research on collaboration behavior.
Participants came from Upwork and were US-based. The 386 sessions cover 10 of 20 O*NET sectors and do not support fine-grained subgroup analysis. One scalar score cannot capture creativity, performance under pressure, or interpersonal communication. Cost is absent from the ranking, and the agents are an early-2026 snapshot.
CollabSkill gives teams a better way to evaluate agents that work with people. Its own ethics statement draws the line for the other half of the leaderboard: the estimated human skill is a research proxy, not a hiring score.
