A few hours can contain many speakers

The launch post describes four speaker-disjoint splits for Indian English and Hindi. Together they contain 4,888 speakers. The public Indian English set has 5.62 hours from 1,444 speakers, while the public Hindi set has 1.33 hours from 468 speakers. Private splits use different speakers and add 5.58 hours of Indian English and 4.47 hours of Hindi.

The collection favors breadth across speakers rather than long recordings from a small cast. Median audio per speaker ranges from 8.28 to 12.39 seconds. The ten largest contributors account for only 2.8% to 6.8% of the duration, depending on the split.

Each speaker also carries 12 attributes, including age, education, occupation, device information, and location. Those fields do not prove a system is fair. They make disparities measurable. A benchmark with only audio and a transcript can report an average error rate. It cannot tell a team whether that average hides a failure concentrated in one region, age group, or handset class.

The aggregate score nearly declared a tie

On the public Indian English set, eight models landed between 4.81 and 4.99 word error rate, a spread of 0.18 points. At corpus level, the models were effectively tied within what the five-hour sample could resolve.

Regional slices were less reassuring. openai/whisper-large-v3-turbo varied by 0.46 points across five zonal-council regions. mistralai/Voxtral-Mini-3B-2507, only 0.14 points behind it on the full corpus, varied by 1.68 points, from 4.38 in the Central zone to 6.06 in the East. The model with the worst result also changed by region: IBM Granite was worst in the North, Microsoft VibeVoice in the South, and Voxtral in the East.

The authors present this as an illustration of what the metadata permits, not a settled finding about regional performance. The zones are coarse rollups of 428 districts, and five hours cannot support sweeping claims. Even with that limitation, the example exposes a practical risk. Two products can look interchangeable in an aggregate table while producing very different error rates for particular users.

Our earlier look at CAR-bench made a related point for coding agents: an average pass rate can hide whether the same system succeeds repeatedly. Monsoon shows the speech version. A rank without information about who receives the errors is incomplete.

Hindi breaks the single-reference assumption

Standard word error rate compares a model's transcript with one reference string. That is manageable when spelling conventions are relatively bounded and a normalizer can map common variants. Conversational Hindi is harder. Code-mixed English words may have several accepted Devanagari spellings, and compounds can be joined or separated without changing what a listener heard.

A model can recognize the audio correctly and still lose points for disagreeing with one annotator's orthography. Different models may prefer different valid spellings, so the chosen reference can even change their order.

The Hindi sets use transcript lattices instead. Each relevant span carries a set of accepted written forms. Candidate variants come from multiple ASR outputs and language-model expansion. Native-speaker linguists then decide which forms are valid for that utterance. The leaderboard scores these lattices with Orthographically-Informed Word Error Rate, or OIWER, rather than ordinary WER.

The OIWER paper reports why that distinction matters across 22 Indian languages. Compared with conventional WER, its framework reduced pessimistic error rates by 6.3 points on average for the tested Canary model. It reduced an apparent Gemini-to-Canary gap from 18.1 to 11.5 points and aligned more closely with human-perceived error than the WER-SN normalization method by 4.9 points. Those results come from the paper's IndicVoices experiments, not the new Monsoon leaderboard.

The Monsoon team also rescored identical Hindi hypotheses against a flattened single reference. Error rates rose for every system, but not uniformly, and two pairs of systems reversed order. The audio and model outputs had not changed. Only the definition of a correct written answer had changed.

A practical evaluation pattern

The leaderboard update puts Indian English in the default English column set, so it contributes to the headline average. Hindi appears in the multilingual tab, where models rank only when they cover every selected language. Public Monsoon splits are available for self-scoring. Different-speaker private splits limit direct optimization against the full test set. The public Indian English and Hindi datasets are published under CC BY 4.0.

An internal speech evaluation can borrow that structure:

1. Keep aggregate WER, but report slices that matter for the deployment population beside it.
2. Separate public development data from a speaker-disjoint private test set.
3. Audit whether the reference format penalizes valid spelling, code mixing, punctuation, or segmentation choices.
4. Report uncertainty when small score gaps are below what the sample can resolve.
5. Keep model outputs and scoring code so a metric change can be replayed without rerunning inference.

That replayable record catches mistakes at the scoring layer. RuBench showed how a benchmark can accidentally score a fallback model rather than the requested one. Monsoon exposes another category error: scoring the requested system with a reference that confuses writing preference with recognition accuracy. Both can produce precise decimals that answer the wrong question.

Hindi changed the leaderboard because the existing evaluation contract was inadequate. Adding another language to the same table would have expanded coverage while preserving a metric that could mis-rank systems. The benchmark became more useful when the new language was allowed to change it.