qhjqhj00 wsj-timit-speech-recognition-eval
wav2vec: Unsupervised Pre-training for Speech Recognition — Schneider et al. (2019) (arXiv:1904.05862, 2019)
What this evaluates
Evaluates speech recognition models on their ability to accurately transcribe spoken audio into text (WER) and characters (LER). It probes the effectiveness of unsupervised pre-training on raw audio for downstream acoustic modeling and decoding.
Datasets
- TIMIT — total ?; splits: train (-1), dev (-1), test (-1)
- WSJ — total 81; splits: si284 (-1), nov93dev (-1), nov92 (-1)
Metrics
WER(primary) — range: percent- Word Error Rate: (Substitutions + Deletions + Insertions) / Total Reference Words. Expressed as a percentage.
LER— range: percent- Letter Error Rate: (Substitutions + Deletions + Insertions) / Total Reference Letters. Expressed as a percentage.
Input / output format
Input: 80-dimensional log-mel filterbank coefficients extracted from raw audio using a 25ms window with 10ms stride, or pre-trained contextual embeddings.
Output: Sequence of words or characters decoded via beam search, optimized using acoustic model probabilities, language model scores, word penalty, and silence penalty.
Scoring recipe
def compute_wer(reference, hypothesis):
ref_words = reference.split()
hyp_words = hypothesis.split()
if not ref_words:
return 0.0
dist = levenshtein_distance(ref_words, hyp_words)
return (dist / len(ref_words)) * 100
Common pitfalls
- Language model hyperparameters (α, β, γ) and beam search settings are tuned separately for word-based vs character-based LMs, significantly affecting final scores.
- Evaluation splits for WSJ are nov92 (test) and nov93dev (validation), which differ from the standard WSJ 0.38/0.92 hour sets used in other benchmarks.
- Pre-training involves cropping audio sequences, removing ~25% of training data, which can impact downstream performance if not accounted for.
Evidence (verbatim from paper)
Final models are evaluated in terms of both word error rate (WER) and letter error rate (LER).
Citation
@misc{schneider2019wav2vec,
title={wav2vec: Unsupervised Pre-training for Speech Recognition},
author={Schneider et al. (2019)},
year={2019},
note={arXiv:1904.05862}
}
- arXiv: 1904.05862