Deepspeech Wer Eval

deepspeech-wer-eval

Deep Speech: Scaling up end-to-end speech recognition — Awni Hannun et al. (2014) (arXiv:1412.5567, 2014)

What this evaluates

This evaluation probes an end-to-end speech recognition system's ability to accurately transcribe conversational telephone speech and robustly handle background noise without phoneme-level modeling or explicit speaker adaptation. It measures transcription accuracy against ground truth references using standard error rates.

Datasets

• Switchboard Hub5’00 (LDC2002S23) — total ?; splits: SWB (-1), CH (-1), Full (-1)
• Custom Noisy Speech Test Set — total 200; splits: clean (100), noisy (100)

Metrics

• word error rate (WER) (primary) — range: percent
  • Calculated as (S + D + I) / N * 100, where S=substitutions, D=deletions, I=insertions, and N=number of words in the reference transcription. Standard Levenshtein-based alignment is used.

Input / output format

Input: Log filter bank spectrograms (80 or 160 channels) + energy term, computed over 20ms windows with 10ms stride. Audio is resampled to 8kHz or 16kHz. Per-speaker or per-utterance normalization is applied.

Output: Word-level transcription string generated by decoding character-level probability vectors with a language model.

Scoring recipe

def compute_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        edits = levenshtein_distance(ref.split(), pred.split())
        total_errors += edits
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0

Common pitfalls

• Researchers often split Hub5’00 into 'easy' (Switchboard) and 'hard' (CallHome) subsets and report results only on the easier portion, whereas this protocol requires evaluating on the full combined set.
• When comparing against commercial APIs, the evaluation only includes utterances where all systems returned a non-empty result, which artificially inflates accuracy compared to penalizing API failures as 100% error.

Evidence (verbatim from paper)

The predicted probability vectors and language model are then fed into our decoder to yield a word-level transcription, which is compared with the ground truth transcription to yield the word error rate (WER).

Citation

@misc{hannun2014deepspeech,
  title={Deep Speech: Scaling up end-to-end speech recognition},
  author={Awni Hannun et al. (2014)},
  year={2014},
  note={arXiv:1412.5567}
}

• arXiv: 1412.5567