Librispeech Eval

librispeech-eval

Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction — Shi et al. (2023) (arXiv:2310.02720, 2023)

What this evaluates

Evaluates speech recognition performance under varying amounts of labeled data (1h, 10h, 100h) and different model sizes. It probes the ability of self-supervised speech models to adapt to downstream transcription tasks with limited supervision.

Datasets

• LibriSpeech — total ?; splits: dev-clean (-1), dev-other (-1), test-clean (-1), test-other (-1); HF librispeech_asr

Metrics

• Word Error Rate (WER) (primary) — range: percent
  • Percentage of words incorrectly recognized compared to the reference transcript. Calculated as (Substitutions + Deletions + Insertions) / Total Words. Lower is better.

Input / output format

Input: Audio waveform and reference transcript.

Output: Predicted word sequence.

Scoring recipe

def compute_wer(predictions, references):
    # Align predicted and reference word sequences
    # Count substitutions (S), deletions (D), insertions (I)
    # N = total words in reference
    wer = (S + D + I) / N * 100
    return wer

Common pitfalls

• Evaluating on different labeled data subsets (1h, 10h, 100h) changes the fine-tuning regime, not just the test set.
• Results are reported with and without a 4-gram language model joint decoding, which significantly impacts WER and must be specified.

Evidence (verbatim from paper)

We evaluate the proposed methods using a variety of speech processing tasks, segmented into four key categories: speech recognition on the LibriSpeech benchmarks... Our chosen evaluation metric is the Word Error Rate (WER).

Citation

@misc{shi2023multiresolutionhubert,
  title={Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction},
  author={Shi et al. (2023)},
  year={2023},
  note={arXiv:2310.02720}
}

• arXiv: 2310.02720