Asr4real Eval

asr4real-eval

ASR4REAL: An extended benchmark for speech models — Riviere et al. (2021) (arXiv:2110.08583, 2021)

What this evaluates

Evaluates automatic speech recognition (ASR) models on their robustness and fairness across diverse real-world conditions, including accented speech, rehearsed speech, and spontaneous conversational speech. It specifically probes performance disparities related to speaker accent, gender, and socio-economic background.

Datasets

• ALLSSTAR — total ?; splits: test (-1)
• NISP — total ?; splits: test (-1)
• VoxPopuli — total ?; splits: test (-1)
• Buckeye — total ?; splits: test (-1)
• CORAAL — total ?; splits: test (-1)

Metrics

• WER (primary) — range: [0, 100] percent
  • Word Error Rate computed as standard edit distance: WER = (S + D + I) / N, where S, D, I are substitutions, deletions, and insertions relative to the reference transcript, and N is the reference word count. The paper reports the median and P90 of the per-speaker WER distribution rather than the mean.

Input / output format

Input: Raw audio recordings of speech (covering various accents, rehearsal levels, and conversational settings) paired with ground-truth transcriptions.

Output: Text transcription generated by the ASR model for each audio clip.

Scoring recipe

def compute_wer(pred, ref):
    # Standard Levenshtein-based WER calculation
    return (substitutions + deletions + insertions) / len(ref.split())

def evaluate(dataset, predictions, references):
    speaker_wers = {}
    for speaker_id, preds, refs in dataset:
        speaker_wers[speaker_id] = compute_wer(preds, refs)
    # Normalize by speaker (already per-speaker)
    all_wers = list(speaker_wers.values())
    median_wer = np.median(all_wers)
    p90_wer = np.percentile(all_wers, 90)
    return median_wer, p90_wer

Common pitfalls

• Reporting mean WER instead of median/P90, which masks outlier sensitivity and fails to capture performance dispersion across speakers.
• Ignoring speaker-level normalization, which skews aggregate statistics when the test set contains imbalanced speaker distributions.
• Assuming standard language models (trained on read speech like Librispeech) transfer effectively to conversational or accented speech without careful parameter tuning or domain adaptation.

Evidence (verbatim from paper)

Tu fully evaluate the robustness and the performances of a model on a given dataset, computing the mean value of the word error rate (WER) is not enough. Indeed, this value is sensitive to outliers and does not give us any information on the dispersion of the WER distribution. We therefore decided to consider the median and the last decile $ ext{P}_{90}$ of the distribution. Furthermore, since the speaker distribution is not always balanced in our test dataset, we always normalize the WER distribution by speaker before estimating any statistic on it.

Citation

@misc{riviere2021asr4real,
  title={ASR4REAL: An extended benchmark for speech models},
  author={Riviere et al. (2021)},
  year={2021},
  note={arXiv:2110.08583}
}

• arXiv: 2110.08583