qhjqhj00 voice-of-india-eval
Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India — Bhogale et al. (2026) (arXiv:2604.19151, 2026)
What this evaluates
Evaluates automatic speech recognition (ASR) systems on real-world, unscripted telephonic conversations across 15 Indian languages. It probes geographic, demographic, and audio quality disparities in model performance, particularly focusing on code-mixed speech and natural orthographic variations.
Datasets
- Voice of India — total 36691; splits: test (-1)
Metrics
Word Error Rate (WER)(primary) — range: percent- Standard WER = (S + D + I) / N, where S=substitutions, D=deletions, I=insertions, N=reference words. This benchmark computes WER against a lattice of multiple valid transcripts to account for orthographic variation, code-mixing, and disfluencies.
Input / output format
Input: Audio recordings of spontaneous telephonic conversations, segmented into utterances via WebRTC VAD.
Output: Text transcript of the spoken audio.
Scoring recipe
def compute_lattice_wer(predictions, lattices):
total_wer = 0.0
for pred, lattice in zip(predictions, lattices):
# Find reference path in lattice that minimizes edit distance to pred
best_ref = min(lattice.paths, key=lambda r: edit_distance(pred, r))
total_wer += wer_score(pred, best_ref)
return (total_wer / len(predictions)) * 100
Common pitfalls
- Standard single-reference WER unfairly penalizes valid code-mixed words and natural spelling variations; evaluators must use the provided lattice/multi-reference format.
- Audio quality and demographic stratification cause significant performance variance by district and device; reporting only aggregate WER masks critical real-world disparities.
- Disfluencies and half-words are optional in the lattice; models that rigidly enforce or ignore these will be mis-scored if evaluated against a flat reference.
Evidence (verbatim from paper)
It uses multiple valid transcripts to mitigate WER penalties for natural spelling variation and code-mixed speech, and conducts granular geographic, demographic, and audio quality analysis revealing significant performance disparities at the district level—highlighting real-world limitations of current ASR systems in India.
Citation
@misc{bhogale2026voiceofindia,
title={Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India},
author={Bhogale et al. (2026)},
year={2026},
note={arXiv:2604.19151}
}
- arXiv: 2604.19151