Shrutilipi Asr Eval

shrutilipi-asr-eval

Effectiveness of Mining Audio and Text Pairs from Public Data for Improving ASR Systems for Low-Resource Languages — Bhogale et al. (2022) (arXiv:2208.12666, 2022)

What this evaluates

Evaluates the quality, diversity, and downstream effectiveness of the Shrutilipi audio-text dataset for low-resource Indian language ASR. It measures how adding mined data improves Word Error Rate (WER) on standard and noisy benchmarks compared to existing datasets.

Datasets

• Shrutilipi — total 6400; splits: train (-1)
• MUCS — total ?; splits: train (-1), blind (-1)
• Kathbath — total ?; splits: test_unknown (-1), test_known (-1)
• CommonVoice — total ?; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate, calculated as the minimum number of insertions, deletions, and substitutions of words required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference.

Input / output format

Input: Raw audio waveform paired with reference text transcript.

Output: Predicted text transcript (decoded via beam search with language model).

Scoring recipe

def compute_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        edit_dist = levenshtein_distance(pred.split(), ref.split())
        total_errors += edit_dist
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0

Common pitfalls

• The paper uses a custom beam-search decoder with specific hyperparameters (α=2, β=-1, beam size=128) and a 6-gram LM; changing these will alter WER significantly.
• The 'hard' benchmark adds background noise from ESC at random SNRs between 3-30 dB; failing to replicate this exact noise injection protocol will invalidate robustness comparisons.
• Shrutilipi's quality is thresholded at Δ≥0.8, but high-quality subset uses Δ>0.95; mixing these thresholds changes training data volume and results.

Evidence (verbatim from paper)

We evaluate performance of Wav2Vec models on the blind set of MUcs (Diwan et al., 2021) and Test Unknown set of Kathbath (Javed et al., 2022a), as shown in Table 4. The model for Bengali was trained on the OpenSLR (Shetty and Umesh, 2021) train set, while we use the MUcs train set for other languages, denoted by E (Existing). For the MUcs blind set, the average WER drops from 17.3% to 15.5%. For Kathbath too, we see a large improvement of 5.8% WER on average.

Citation

@misc{bhogale2022shrutilipi,
  title={Effectiveness of Mining Audio and Text Pairs from Public Data for Improving ASR Systems for Low-Resource Languages},
  author={Bhogale et al. (2022)},
  year={2022},
  note={arXiv:2208.12666}
}

• arXiv: 2208.12666