title: "OmniaBench's near-tied agents differ by 38 points on the work they can do" slug: "omniabench-overall-score-hides-agent-skill-gaps" excerpt: "OmniaBench shows how one general-agent score can hide large differences in planning, optimization, and tool-use behavior." category: "AI Research" tags:
- "OmniaBench"
- "AI agents"
- "benchmarks"
- "open models"
- "model evaluation" author: "ISH Team" reading_time: 7
Three agents land within one third of a percentage point on a new general-agent benchmark. Claude Sonnet 5 scores 58.54%, GPT-5.6-Sol scores 57.14%, and the open-source GLM-5.2 scores 56.83%. Read only that final column and they look interchangeable.
They are not. On OmniaBench's optimization tasks, GPT-5.6-Sol scores 65.00% while GLM-5.2 scores 26.67%, a difference of 38.33 percentage points. On the benchmark's derived single-turn tasks, GLM-5.2 leads the same GPT model 69.00% to 59.00%.
Models having different strengths is no surprise. The surprise is how thoroughly the overall score conceals them.
One benchmark, four different kinds of work
OmniaBench is a July 2026 benchmark for general agents built by researchers from Huawei Cloud, Peking University, Renmin University of China, Beijing Institute of Technology, Tsinghua University, and Zhongguancun Academy. Its full collection has 1,431 tasks. The released challenging set contains 644 tasks across consumer, business, and enterprise scenarios.
The tasks follow four construction routes. DAG covers multi-turn, stateful tool chains. DAG-S reshapes related scenarios into more detailed single-turn requests. Solver covers selection, scheduling, allocation, and optimization. Program requires procedural reasoning with branches, iteration, execution, and debugging.
The mix is uneven: 354 DAG tasks, 200 DAG-S tasks, 60 Solver tasks, and 30 Program tasks. The overall Pass@1 score is simply the share of all 644 tasks completed. It is therefore shaped most heavily by the two largest routes.
The authors do not hide this weighting. The official project page publishes route-level results so readers can inspect each profile. Information disappears only when someone copies the overall rank and leaves those columns behind.
The open model's near tie is real and incomplete
GLM-5.2 finishing 0.31 points behind GPT-5.6-Sol is notable. So is DeepSeek-V4-Pro's 54.50% result, which places it near several closed models. The open repository includes a model-agnostic evaluation harness, scoring pipeline, taxonomy files, and task-domain maps. That makes the benchmark more useful than a static screenshot.
Calling GLM-5.2 "near frontier overall" leaves out the important part. It scores 54.80% on DAG, 26.67% on Solver, 60.00% on Program, and 69.00% on DAG-S. GPT-5.6-Sol scores 55.37%, 65.00%, 50.00%, and 59.00% on those same routes. Different capability shapes happen to produce similar totals.
Claude Sonnet 5, the overall leader, has another profile: 57.34% on DAG, 56.67% on Solver, 63.33% on Program, and 60.50% on DAG-S. It also averages 64.16 tool-call steps, compared with 38.73 for GPT-5.6-Sol and 57.08 for GLM-5.2. Success and execution length are separate procurement questions.
If your agent mainly schedules resources, the Solver column matters more than the overall rank. If it translates a detailed request into a single tool-driven run, DAG-S may be closer to the workload. A model can be the right choice without being the benchmark winner.
Why the task mix changes the answer
An aggregate score embeds a product decision. Giving 354 stateful DAG tasks and 30 Program tasks equal weight per task means the former contributes almost twelve times as much to the final number. A hospital operations team, developer-tools company, and consumer assistant may reasonably choose three different weightings.
The benchmark also reports business, consumer, and enterprise splits, plus ten capability dimensions and eight atomic difficulty factors. These views turn one ranking into a profile. They are more work to read, but they expose whether a result comes from task understanding, information gathering, planning, state management, tool use, or another capability.
The leaderboard answers a specific question: which model did best on this published mixture under this published harness? Treating that answer as a shopping list skips the question of whether production contains the same work.
Build a workload-weighted scorecard
The weights can come from production traces. Sample completed and failed requests, remove sensitive data, and label each by the closest task route. Estimate how much work is multi-turn and stateful, optimization-heavy, procedural, or detailed and single-turn. Equal weights make sense only when production is actually balanced.
Some requirements belong outside the weighted average. A customer-support agent might need reliable state updates and strict constraint adherence even if those gates reduce its average score. A coding workflow might care about reproducibility across repeated runs, the issue raised in our article on CAR-bench and Pass@3. A model that wins once but varies widely may be harder to operate.
A small evaluation can then run behind the same interface. Keep prompts, tools, retry limits, timeouts, and reasoning settings fixed. A shared endpoint such as api.ish.chat can simplify model switching, but the evaluation record still needs exact model IDs and parameters. Our guide to grading the full agent trajectory explains why the final answer alone is insufficient.
Cost and behavior belong beside correctness. OmniaBench allows up to 200 tool-call steps and reports averages that vary substantially by model. More calls can mean useful exploration, or redundant work and recovery from errors. Review samples before interpreting the count. The practical interface advice in designing agent tools for reviewable results applies here: observable state makes both evaluation and debugging easier.
Write down the weighting formula. If Solver tasks count for 50% of the decision, say so. The resulting rank will not be universal, but colleagues can audit it and connect it to the work that needs doing.
Limits worth keeping visible
OmniaBench's environments are synthesized from a taxonomy built with web research and human refinement. Multi-turn tasks use a model as the user simulator, and most routes use rubric-based model judging; Program uses executable VerifyCode instead. The paper reports judge-substitution and user-simulator stability tests, but synthetic environments and model judges remain different from live users and production systems.
The evaluations ran from June 29 to July 15, 2026 with specific APIs, reasoning settings, and retry rules. Future model versions can move the numbers. The released 644-task dataset is also a curated challenging subset, not a random sample of all agent work.
Three nearly tied totals can describe three meaningfully different tools. That result survives the caveats. An overall score is a useful first look, but a poor place to end a product decision.
