qhjqhj00 voice-search-wer-eval
Transformer Transducer: One Model Unifying Streaming and Non-streaming Speech Recognition — Tripathi et al. (2020) (arXiv:2010.03192, 2020)
What this evaluates
Evaluates speech recognition accuracy and latency trade-offs for streaming vs. non-streaming decoding on a proprietary voice-search dataset. It probes the model's ability to maintain low word error rate while minimizing output delay and computational overhead.
Datasets
- Voice Search — total ?; splits: test (14000)
Metrics
WER(primary) — range: percent- Word Error Rate: the percentage of words incorrectly recognized relative to the reference transcript.
RTF— range: other- Real-Time Factor: the ratio of processing time to audio duration.
Alignment Delay— range: other- Mean difference in word alignment timestamps between reference and model: D = (1/N) * sum(T_ref - T_Y).
Input / output format
Input: 30ms logmel energy features (128-dim, stacked 4x, subsampled 3x) derived from speech waveforms using a 32ms window with 10ms shift.
Output: Transcribed text sequence and word alignment timestamps.
Scoring recipe
def compute_wer(predictions, references):
errors = sum(edit_distance(p, r) for p, r in zip(predictions, references))
total = sum(len(r) for r in references)
return 100 * errors / total
def compute_alignment_delay(ref_times, model_times):
return sum(t_ref - t_model for t_ref, t_model in zip(ref_times, model_times)) / len(ref_times)
Common pitfalls
- Confusing fixed lookahead latency (e.g., 240ms buffer) with actual word alignment delay, which varies by model configuration.
- Reporting RTF without specifying hardware (TPU vs CPU) and batch size, as inference speed heavily depends on these factors.
- Overlooking that constrained alignment training improves delay but degrades low-latency WER.
Evidence (verbatim from paper)
Table 1 shows WER and benchmarking results for different label encoder architectures.
Citation
@misc{tripathi2020transformertransducer,
title={Transformer Transducer: One Model Unifying Streaming and Non-streaming Speech Recognition},
author={Tripathi et al. (2020)},
year={2020},
note={arXiv:2010.03192}
}
- arXiv: 2010.03192