Deep Speech 2 Eval

deep-speech-2-eval

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin — Amodei et al. (arXiv:1512.02595, 2015)

What this evaluates

Evaluates end-to-end speech recognition accuracy across diverse acoustic conditions including clean read speech, accented speech, and noisy speech in English and Mandarin. It benchmarks model performance against both automated baselines and human transcribers to measure real-world applicability.

Datasets

• WSJ eval'92 — total ?; splits: test (-1)
• WSJ eval'93 — total ?; splits: test (-1)
• LibriSpeech test-clean — total ?; splits: test (-1)
• LibriSpeech test-other — total ?; splits: test (-1)
• VoxForge Accented Speech — total 4096; splits: test (4096)
• CHiME eval clean — total 1320; splits: test (1320)
• CHiME eval real — total 1320; splits: test (1320)
• CHiME eval sim — total 1320; splits: test (1320)
• Baidu internal English test — total 3300; splits: test (3300)
• Baidu internal Mandarin dev — total 2000; splits: dev (2000)
• Baidu internal Mandarin test — total 1882; splits: test (1882)

Metrics

• WER (primary) — range: percent
  • Word Error Rate calculated as (Substitutions + Deletions + Insertions) / Total Reference Words * 100. Measures the edit distance between predicted and reference transcriptions at the word level.

Input / output format

Input: Raw audio waveform recordings, typically ~5 seconds per clip.

Output: Predicted text transcription string corresponding to the spoken utterance.

Scoring recipe

def compute_wer(predictions, references):
    total_words = sum(len(ref.split()) for ref in references)
    errors = 0
    for pred, ref in zip(predictions, references):
        errors += levenshtein_distance(pred.split(), ref.split())
    return (errors / total_words) * 100

Common pitfalls

• Human baseline is derived from the better of two independent MTurk transcriptions per clip, not a single annotator or majority vote.
• CHiME results are computed using only a single audio channel per utterance, ignoring the multi-channel setup that could improve performance.
• No model adaptation or hyperparameter tuning is performed per test set; decoding parameters are fixed on a held-out development set.

Evidence (verbatim from paper)

We obtain a measure of human level performance by paying workers from Amazon Mechanical Turk to hand-transcribe all of our test sets. Two workers transcribe the same audio clip, that is typically about 5 seconds long, and we use the better of the two transcriptions for the final WER calculation. They are free to listen to the audio clip as many times as they like. These workers are mostly based in the United States, and on average spend about 27 seconds per transcription. The hand-transcribed results are compared to the existing ground truth to produce a WER.

Citation

@misc{amodei2015deepspeech2,
  title={Deep Speech 2: End-to-End Speech Recognition in English and Mandarin},
  author={Amodei et al.},
  year={2015},
  note={arXiv:1512.02595}
}

• arXiv: 1512.02595