Yodas Speech Recognition Eval

yodas-speech-recognition-eval

YODAS: Youtube-Oriented Dataset for Audio and Speech — Li et al. (2024) (arXiv:2406.00899, 2024)

What this evaluates

Evaluates monolingual automatic speech recognition performance across multiple languages using a large-scale, YouTube-collected audio dataset. It probes the model's ability to accurately transcribe spoken language from noisy, real-world video subtitles after alignment filtering.

Datasets

• YODAS — total ?; splits: train (1000000), test (1000)

Metrics

• CER (primary) — range: percent
  • Character Error Rate, calculated as the minimum number of character edits (insertions, deletions, substitutions) required to transform the predicted transcript into the reference transcript, divided by the number of characters in the reference transcript.

Input / output format

Input: Raw audio waveform paired with a reference transcript (subtitle) for a specific language.

Output: Greedy decoded character sequence (transcript) from the model.

Scoring recipe

def compute_cer(predictions, references):
    total_errors = 0
    total_ref_chars = 0
    for pred, ref in zip(predictions, references):
        total_errors += edit_distance(pred, ref)
        total_ref_chars += len(ref)
    return (total_errors / total_ref_chars) * 100

Common pitfalls

• Failing to apply the CTC alignment score threshold (>2.0) before training/evaluation, which includes heavily misaligned audio-text pairs and inflates error rates.
• Using automatic subtitles instead of manual ones without accounting for the significantly higher deletion error rate (26.6% vs 7.0%) in the automatic subset.
• Applying a uniform BPE vocabulary size across all languages, which fails to account for the higher character complexity of CJK scripts and artificially worsens CER.

Evidence (verbatim from paper)

Table.[5] displays the testing outcomes for the top-15 languages, measured by the Character Error Rate (CER). The respective CER for each language spans from 6 to 15. The best performance is recorded for Hungarian, with a CER of 6.2, while Japanese exhibits the least performance with a CER of 14.7.

Citation

@misc{li2024yodas,
  title={YODAS: Youtube-Oriented Dataset for Audio and Speech},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2406.00899}
}

• arXiv: 2406.00899