Vaani Asr Lid Eval

vaani-asr-lid-eval

VAANI: Capturing the language landscape for an inclusive digital India — Pulikodan et al. (2026) (arXiv:2603.28714, 2026)

What this evaluates

This evaluation protocol assesses the utility of the Vaani dataset for fine-tuning automatic speech recognition (ASR) and spoken language identification (LID) models across diverse Indian languages and regions. It measures performance gains from fine-tuning on Vaani's transcribed audio and images against established benchmarks, highlighting regional dialectal variations and low-resource language capabilities.

Datasets

• Vaani — total ?; splits: train (-1), val (-1), test (-1)
• FLEURS — total ?; splits: test (-1)
• Kathbath — total ?; splits: test (-1)

Metrics

• Word Error Rate (WER) (primary) — range: percent
  • The percentage of words incorrectly recognized (substitutions, deletions, insertions) relative to the total number of words in the reference transcription. Lower values indicate better performance.
• Accuracy — range: percent
  • The percentage of correctly predicted language labels out of the total number of test instances. Per-language f1-scores are also reported as percentages.

Input / output format

Input: Audio recordings of spontaneous speech (and corresponding images for multimodal tasks), with ground-truth transcriptions for ASR and language labels for LID.

Output: Predicted word sequences (ASR) or discrete language class labels (LID).

Scoring recipe

def compute_wer(predictions, references):
    total_words = sum(len(ref.split()) for ref in references)
    errors = 0
    for pred, ref in zip(predictions, references):
        errors += edit_distance(pred.split(), ref.split())
    return (errors / total_words) * 100

def compute_accuracy(predictions, labels):
    correct = sum(1 for p, l in zip(predictions, labels) if p == l)
    return (correct / len(labels)) * 100

Common pitfalls

• Evaluating low-resource languages (e.g., Chakma, Bhojpuri) only on Vaani due to the absence of other public benchmarks, which limits cross-dataset comparability.
• Forcing output normalization on models like Gemma-3n-2B to remove repeated words before computing WER, which may artificially inflate scores compared to standard ASR evaluation pipelines.
• Region-specific fine-tuning results show strong geographic proximity bias, meaning performance drops significantly for distant states even within the same language, which can mislead generalization claims.

Evidence (verbatim from paper)

We fine-tuned the Whisper-small model (244M parameters)Radford et al. (2022) separately for Hindi (331 hours), Kannada (80.244 hours), Telugu (69.039 hours), and Bengali (101 hours) on a subset of the Vaani dataset, and evaluated each model independently on multiple benchmark datasets ... We normalized the output of Gemma-3n-2B when computing WER for the non-finetuned model to remove repeated words.

Citation

@misc{pulikodan2026vaani,
  title={VAANI: Capturing the language landscape for an inclusive digital India},
  author={Pulikodan et al. (2026)},
  year={2026},
  note={arXiv:2603.28714}
}

• arXiv: 2603.28714