qhjqhj00 librispeech-voice-search-wer-eval
Parallel Rescoring with Transformer for Streaming On-Device Speech Recognition — Li et al. (2020) (arXiv:2008.13093, 2020)
What this evaluates
Evaluates the word error rate of streaming speech recognition systems using a first-pass RNN-T model followed by a second-pass rescorer. It probes the ability of parallel Transformer rescoring to improve transcription accuracy while maintaining low-latency streaming constraints on-device.
Datasets
- Librispeech — total ?; splits: train (-1), test-clean (-1), test-other (-1)
- Google Voice Search — total 14000; splits: test (14000)
Metrics
WER(primary) — range: percent- Standard Word Error Rate: (S + D + I) / N * 100%, where S=substitutions, D=deletions, I=insertions, and N=number of words in the reference transcription.
Input / output format
Input: Acoustic features and top-4 hypothesis sequences from a first-pass RNN-T model.
Output: Rescored probability distributions over 4096 word pieces, yielding a final transcription hypothesis.
Scoring recipe
def wer(predictions, references):
# predictions and references are lists of word strings
# Align predictions to references using standard edit distance
edit_dist = levenshtein_distance(predictions, references)
ref_len = len(references)
if ref_len == 0:
return 0.0
return (edit_dist / ref_len) * 100.0
Common pitfalls
- Using causal self-attention during training with ground truth makes the task trivial; random token swapping (15%) is required for CE training.
- N-best size for MWER training must exactly match evaluation (top 4 hypotheses).
- Evaluation must use Exponential Moving Average (EMA) weights, not the final training checkpoint.
Evidence (verbatim from paper)
As is shown in Table 1, both the LSTM rescorer and the Transformer rescorer significantly improve the WER of the clean and noisy test sets compared to the RNN-T only model with 10-20% relative improvement, alleviating the limited context problem for the 1st-pass model while still maintaining low-latency streaming recognition. The Transformer rescorer further improves the WER slightly over the LSTM rescorer, and also significantly reduce the 2nd-pass latency, which is studied in detail in Section 4.
Citation
@misc{li2020parallelrescoring,
title={Parallel Rescoring with Transformer for Streaming On-Device Speech Recognition},
author={Li et al. (2020)},
year={2020},
note={arXiv:2008.13093}
}
- arXiv: 2008.13093