What BenchMIRT adds

BenchMIRT draws on item response theory, a set of methods from psychometrics. A normal benchmark average counts correct answers. Item response theory also estimates the difficulty of each question and whether it distinguishes strong performers from weak ones.

Ai2 uses a multidimensional version of the method. It does not assume every item measures one shared ability. BenchMIRT recovers latent dimensions from patterns in model responses, then estimates an ability profile for each model and difficulty and discrimination values for each item.

The researchers analyzed 100 open-weight language models across 16 benchmarks and more than 34,000 questions. Six benchmarks targeted general reasoning; ten came from the Olmo 3 safety suite. BenchMIRT was not told which benchmarks belonged to which category. Repeated runs still recovered two dominant dimensions, which the authors interpreted as safety and general reasoning.

Ai2 released the code under Apache 2.0 and published the related datasets in a Hugging Face collection. An independent audit needs the item-level response matrix, though. The final averages on a leaderboard do not contain enough information.

When a safety test is partly a comprehension test

Many results matched their labels. MMLU-Pro, BBH, and GPQA aligned mainly with reasoning. HarmBench, StrongReject, and several jailbreak tests aligned mainly with safety.

BBQ tests whether a model falls back on social stereotypes. BenchMIRT found a correlation of 0.85 with the recovered reasoning dimension and -0.06 with safety. Some BBQ questions require a model to keep track of people and resolve ambiguity before bias can be measured. A wrong answer might come from stereotyping, poor comprehension, or both.

WMDP poses another problem. Its score correlated at -0.89 with reasoning and 0.21 with safety. WMDP rewards the absence or refusal of dangerous dual-use knowledge, so a more capable model can receive a worse desired score. The negative direction is part of the measurement, not a typo. Describing WMDP as a pure safety score conceals the capability tradeoff.

Different sections of one test can split as well. Ai2 reports that HarmBench's harmful and contextual prompts mostly followed the safety dimension, while its copyright questions leaned toward reasoning. WildJailbreak contains 250 benign items that leaned toward reasoning, inside a larger harmful set that followed safety. An average folds those failure modes together and removes the explanation.

Agent evaluations have the same problem. RepoComplianceBench found that coding agents handled extra procedural steps better than hard stop rules. A single pass rate could not show which sort of instruction failed. BenchMIRT supplies a way to search for mixed signals before giving a score a broad name.

Ten percent is a research result, not a discount code

BenchMIRT ranked questions by how much information they carried about the recovered abilities. Across the 16 benchmarks, a selected 10% of items generally preserved nearly the same picture of which models were stronger or weaker. At 50%, the reduced tests often matched the full suite even more closely.

Cheaper evaluations would be useful, but that result does not support throwing away 90% of any test. The subset was selected from response data for this benchmark family and model population. A compact test may preserve a ranking while dropping rare behavior with serious consequences. Item-level analysis could also identify and remove the strongest safety questions, making an unsafe model easier to pass. The researchers identify this as a risk of the method.

The study has two other limits that belong beside every summary of its findings. All 100 models were released by March 2025, as the public model-statistics dataset records. Newer models might yield a different response structure. The dimensions also depend on the set of benchmarks supplied. Another collection of tests might produce different latent abilities.

BenchMIRT predicted held-out item responses with 79% accuracy, compared with 70% for a baseline based on each model's benchmark average. Yet the ordinary average did slightly better when the sole goal was ranking models on randomly held-out items. BenchMIRT provides more detail at question level; it does not replace every use of an average.

A scorecard should keep the disagreements

Most product teams will not fit a multidimensional item response model for every release. They can still stop accepting category names as ground truth.

First write down the decision attached to each score. It may select a model, trigger human review, restrict a feature, or block deployment. Next, divide failures into item families that match the product, including harmful compliance, benign over-refusal, ambiguity, factual knowledge, and instruction following. When a safety result rises and falls with a reasoning result, inspect the examples before crediting the change to safer behavior.

Keep one operational replay suite beside the research benchmarks. Run representative conversations and tool calls with the prompts, policies, and limits used in production. Answer benchmarks do not cover permissions, network access, tool confirmation, or recovery from a partial action. The READY study of review cost makes a related point: reliability depends on what reviewers must inspect and how often.

A shared API layer can make this comparison cleaner. Teams routing the same workload to several providers through api.ish.chat can hold application logic and prompts constant while logging item-level outcomes. The resulting table should say which model failed which task family, under which policy, and what the application did next.

A global average is useful for orientation, but it should not decide a deployment alone. The benchmark name describes its intended category. The response data shows which abilities shaped the score.