Asr Robustness Eval

asr-robustness-eval

Rethinking Evaluation in ASR: Are Our Models Robust Enough? — Likhomanenko et al. (2020) (arXiv:2010.11745, 2020)

What this evaluates

Evaluates the robustness and cross-domain generalization of Automatic Speech Recognition (ASR) models by measuring Word Error Rate (WER) across multiple public and in-house English speech datasets with varying acoustic conditions, sampling rates, and speech types.

Datasets

• LibriSpeech — total ?; splits: train (960), dev-clean (5), dev-other (5), test-clean (5), test-other (5)
• SwitchBoard & Fisher — total ?; splits: train (2300), dev (6), test (4)
• WSJ — total ?; splits: train (81), dev (1), test (1)
• Common Voice — total ?; splits: train (693), dev (27), test (26)
• TED-LIUM v3 — total ?; splits: train (452), dev (2), test (3)
• Robust Video — total ?; splits: train (5000), dev (14), test (75)
• CHiME-6 — total 40; splits: dev (-1), eval (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate computed as (Substitutions + Deletions + Insertions) / Total Words in reference transcript. Evaluated using the standard Kaldi recipe for data processing and alignment.

Input / output format

Input: 16kHz audio resampled from original datasets, processed into 80 log-mel spectrogram features (25ms window, 10ms shift), normalized to zero mean and unit variance per input sequence.

Output: Transcribed text sequence using a fixed token set of 26 English letters, apostrophe, and word boundary token; punctuation removed for WSJ.

Scoring recipe

def compute_wer(reference, hypothesis):
    ref_words = reference.split()
    hyp_words = hypothesis.split()
    edit_dist = levenshtein_distance(ref_words, hyp_words)
    total_words = len(ref_words)
    if total_words == 0:
        return 0.0
    return (edit_dist / total_words) * 100

Common pitfalls

• The reported 'average WER' is calculated as the average of per-dataset average WERs, not a global average across all test words, to prevent dataset size bias.
• Audio must be resampled to 16kHz and features normalized per sequence before evaluation to ensure consistency across datasets with different original sampling rates.
• Punctuation is stripped from transcriptions (e.g., WSJ) and tokenization uses a fixed alphabet set rather than word-pieces, which affects alignment and error counting.

Evidence (verbatim from paper)

Table 4: WER of models evaluated on all datasets (downsampled to 16kHz) with a greedy decoding and no LM (top row), with in-domain n-gram LM beam-search decoding (middle row) and with additional second-pass rescoring by in-domain Transformer LM (below row). Joint models are also decoded with CC LM with either a single-pass (top row) or a two-pass (bottom row) decoding. State-of-the-art (SOTA) models are given from WSJ [Hadian et al., 2018], TEDLIUM [Zhou et al., 2020], LibriSpeech [Gulati et al., 2020], SwitchBoard & Fisher [Han et al., 2017]. The SOTA models are all decoded with in-domain LMs. The average is computed as average of averages for LibriSpeech's validations/tests, and SwitchBoard's tests (SB, CH) sets, so as not to weight them more heavily.

Citation

@misc{likhomanenko2020rethinking,
  title={Rethinking Evaluation in ASR: Are Our Models Robust Enough?},
  author={Likhomanenko et al. (2020)},
  year={2020},
  note={arXiv:2010.11745}
}

• arXiv: 2010.11745