Voice Arena and Hugging Face have released Monsoon, a set of open automatic speech recognition (ASR) evaluation datasets for the Open ASR Leaderboard, aimed at exposing failure modes that a single aggregate metric like WER can hide. Monsoon supplies four speaker‑disjoint splits across two languages — Indian English (en‑IN) and Hindi (hi‑IN) — and focuses on breadth of speakers rather than total hours: the collection covers 4,888 speakers in total and records 12 speaker attributes per clip.
Why this matters
Benchmarks and leaderboards strongly influence which models are developed and iterated. Recent efforts have improved metric trustworthiness (private held‑out splits, benchmark‑fitting analysis, improved normalisers), but a single score still obscures unequal error distributions across demographic groups. Prior work has documented disparities by race, gender, age and accent; Monsoon is intended to make such differences visible on a public leaderboard by varying the test sets along multiple axes: geography, age, gender, vocabulary, devices, acoustic environments, speech type, speech rate and the presence of multiple valid transcripts.
Dataset structure and key numbers
Monsoon consists of four splits: Monsoon en‑IN public and private (Indian English), Monsoon hi‑IN public and private (Hindi). Key statistics:
- Monsoon en‑IN public: 5.62 hours, 1,444 speakers, mean clip 9.6 s (median 10.4 s), M/F 50/50, 428 districts, 24 states/UTs, 556 device models, conversational spontaneous speech, normalized transcripts with disfluencies preserved.
- Monsoon en‑IN private: 5.58 hours, 1,405 speakers, similar metadata.
- Monsoon hi‑IN public: 1.33 hours, 468 speakers, mean clip 6.4 s (median 5.0 s), 202 districts, 11 states/UTs, 315 device models, conversational spontaneous speech; transcriptions supplied as lattices listing accepted orthographic variants.
- Monsoon hi‑IN private: 4.47 hours, 1,571 speakers, mean clip 6.6 s (median 5.3 s), 295 districts, 12 states/UTs, 582 device models, lattice transcriptions.
Audio comes from unscripted two‑person conversations recorded dual‑channel, with clips segmented to a single channel so each clip contains one speaker. Additional per‑clip fields include occupation, education, marital status, income band, handset brand, current city and years in current district. Example en‑IN public clips (with metadata): 29‑year‑old woman, West Tripura, samsung SM‑G781B; 32‑year‑old woman, Satna, Madhya Pradesh, samsung SM‑E146B; 57‑year‑old man, Puducherry, Xiaomi M2006C3LI.
Design principles: many speakers, wide variance
Monsoon emphasizes speaker diversity over duration per speaker. Notable properties:
- Low speaker concentration: the top ten contributors account for 2.8–6.8% of total duration; more than half of speakers contribute exactly one segment.
- Broad geography and device coverage: the en‑IN public set spans 428 native districts; device models range from 315 to 582 with no single model contributing more than 2.1% of segments.
- Accent diversity: Indian English is represented across all six zones. In the en‑IN public set, 35% of segments are from southern speakers, 18% East, 18% Central, 16% North and 11% West.
These properties allow analyses that disaggregate model performance by region, age, device, and other attributes rather than hiding variation in an average WER.
Transcription and handling orthographic variation
References are human‑produced following a pipeline: internal ASR drafts are corrected by native‑speaker linguists under a five‑level protocol with separation of duties and independent verification at each stage. Numerals are spelled out; annotator activity was monitored automatically and segments that failed checks were re‑transcribed.
Hindi presents substantial orthographic variation: code‑mixing, multiple valid Devanagari spellings for English‑origin words and inconsistent compounding mean no single normaliser can collapse variants reliably. To address this, the Hindi splits ship lattices — for each transcript span a set of accepted written forms — and Monsoon reports Orthographically‑Informed Word Error Rate (OIWER), introduced by AI4Bharat. OIWER aligns hypotheses to the set of accepted forms per span so any admitted form counts as correct, charging only genuine recognition errors.
Scoring the same hypotheses against a flattened single reference (picking the first lattice variant for each span) increases error rates and can change model rankings, because a single reference rewards reproducing the annotator’s orthography. Monsoon publishes voi‑oiwer, its OIWER implementation, so results are reproducible.
Collection, recruitment and quality control
Contributors were recruited through the Voice Arena community. Pairs recorded two‑person conversations via a peer‑to‑peer interface, using their own handsets and network connections; low‑end devices and unstable bandwidth were deliberately included to reflect in‑the‑wild conditions. Prospective contributors passed language proficiency screening, were compensated, gave informed consent for use in training and distribution, and per‑speaker duration caps prevented domination by prolific contributors.
Elicitation used open narrative prompts with successive follow‑ups across domains (travel, healthcare, agriculture, education, digital services) to encourage extended, unscripted speech. Candidate topics were generated with large language models and localized by native‑speaker linguists.
Quality control included automatic language identification to verify spoken language, a gender corroboration classifier, detectors for replayed or pre‑recorded audio, SNR estimation to remove unintelligible recordings while preserving realistic background noise, segmentation by voice activity detection, and DNSMOS P.808 checks. Segmentation used two seconds of continuous silence or a fifteen‑second soft cap and was applied independently per channel so every segment is single‑speaker.
Example: regional differences revealed
An illustrative analysis on the public en‑IN split shows eight leaderboard models scoring between 4.81 and 4.99 WER overall — a 0.18 point range. Grouping speakers by geographic zone, however, reveals larger variation: one model varies by 0.46 points across zones, another by 1.68 points (e.g. 4.38 WER in Central vs. 6.06 in East). Two systems that are statistically similar at the corpus level can differ substantially in how their accuracy depends on speaker region. Which zone is hardest also differs by model, indicating model‑specific biases rather than an absolute region difficulty.
Region is one of twelve recorded attributes and the released data contain district‑level fields; a prior closed benchmark showed district‑level error rates ranging roughly 4% to 44% when disaggregated. Monsoon makes this class of analysis possible on a public leaderboard test set.
How to get evaluated
To be evaluated on the private splits, add your model to the Open ASR Leaderboard via the GitHub process (open a pull request and follow the model checklist). Public datasets are available for self‑scoring; Hugging Face will verify public results and compute metrics on the private splits. Indian English appears on the main leaderboard as Voice Arena Monsoon in the default column set and contributes to headline Average WER; the private split populates the aggregated Private (conversational) column with Appen and DataoceanAI. The Hindi public and private sets appear on the Multilingual tab and models are ranked there only if they support every selected language.
Conclusion
Hindi exemplifies broader problems: languages with multiple orthographies and populations under‑sampled by conventional benchmarks produce failure modes that a single WER hides. Monsoon does not fix those issues, but it provides a way to see them on a widely followed leaderboard by supplying speaker‑level metadata and orthography‑aware references. The four Monsoon splits are part of Voice Arena's broader dataset initiative focused on the Global South.
