wenetspeech-eval
WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition — Zhang et al. (2021) (arXiv:2110.03370, 2021)
What this evaluates
Evaluates Mandarin speech recognition systems across diverse, real-world domains (internet, meetings) and mixed Mandarin-English content. It benchmarks the robustness of ASR models against noisy, production-level audio and varying data scales.
Datasets
Metrics
MER% (primary) — range: percent
- Mixture Error Rate (MER) is calculated as the edit distance between predicted and ground truth tokens, where tokens are defined as Mandarin characters and English words, divided by the length of the ground truth sequence, expressed as a percentage.
Input / output format
Input: Audio recordings (typically processed into FBank features, optionally with i-vectors) paired with ground truth transcriptions containing mixed Mandarin characters and English words.
Output: Predicted text transcription.
Scoring recipe
def compute_mer(predictions, references):
# Tokenize both into Mandarin characters and English words
pred_tokens = tokenize_mandarin_english(predictions)
ref_tokens = tokenize_mandarin_english(references)
# Compute Levenshtein edit distance
edits = levenshtein_distance(pred_tokens, ref_tokens)
# Calculate error rate as percentage
mer = (edits / len(ref_tokens)) * 100
return mer
Common pitfalls
- MER treats Mandarin characters and English words as uniform tokens in edit distance, differing from standard Chinese Character Error Rate (CER) or Word Error Rate (WER).
- The paper explicitly states these are baseline results for toolkits (Kaldi, ESPnet, WeNet) and do not reflect state-of-the-art performance.
- Different toolkits use varying feature extraction parameters (e.g., FBank window/shift, i-vectors) and decoding strategies, so results are toolkit-specific rather than purely model-specific.
Evidence (verbatim from paper)
In Table 5, we report the experimental results in Mixture Error Rate (MER)[[40]], which considers Mandarin characters and English words as the tokens in the edit distance calculation, on three designed test sets and one well-known, publicly available test set (i.e. AIShell-1[[41]] test) with Kaldi, ESPNet and WeNet toolkits respectively.
Citation
@misc{zhang2021wenetspeech,
title={WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition},
author={Zhang et al. (2021)},
year={2021},
note={arXiv:2110.03370}
}
1---2name: wenetspeech-eval3description: Evaluates Mandarin speech recognition systems across diverse, real-world domains (internet, meetings) and mixed Mandarin-English content. It benchmarks the robustness of ASR models against noisy, production-level audio and varying data scales. Use when the user wants to benchmark on WenetSpeech, or asks about evaluating this task. Reports MER%.4---56# wenetspeech-eval78> WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition — Zhang et al. (2021) (arXiv:2110.03370, 2021)910## What this evaluates1112Evaluates Mandarin speech recognition systems across diverse, real-world domains (internet, meetings) and mixed Mandarin-English content. It benchmarks the robustness of ASR models against noisy, production-level audio and varying data scales.1314## Datasets1516- **WenetSpeech** — total 22400; splits: train_S (-1), train_M (-1), train_L (-1), dev (-1), test_net (-1), test_meeting (-1); repo https://github.com/wenet-e2e/WenetSpeech1718## Metrics1920- `MER%` **(primary)** — range: percent21 - Mixture Error Rate (MER) is calculated as the edit distance between predicted and ground truth tokens, where tokens are defined as Mandarin characters and English words, divided by the length of the ground truth sequence, expressed as a percentage.2223## Input / output format2425**Input**: Audio recordings (typically processed into FBank features, optionally with i-vectors) paired with ground truth transcriptions containing mixed Mandarin characters and English words.2627**Output**: Predicted text transcription.2829## Scoring recipe3031```python32def compute_mer(predictions, references):33 # Tokenize both into Mandarin characters and English words34 pred_tokens = tokenize_mandarin_english(predictions)35 ref_tokens = tokenize_mandarin_english(references)36 # Compute Levenshtein edit distance37 edits = levenshtein_distance(pred_tokens, ref_tokens)38 # Calculate error rate as percentage39 mer = (edits / len(ref_tokens)) * 10040 return mer41```4243## Common pitfalls4445- MER treats Mandarin characters and English words as uniform tokens in edit distance, differing from standard Chinese Character Error Rate (CER) or Word Error Rate (WER).46- The paper explicitly states these are baseline results for toolkits (Kaldi, ESPnet, WeNet) and do not reflect state-of-the-art performance.47- Different toolkits use varying feature extraction parameters (e.g., FBank window/shift, i-vectors) and decoding strategies, so results are toolkit-specific rather than purely model-specific.4849## Evidence (verbatim from paper)5051> In Table 5, we report the experimental results in Mixture Error Rate (MER)[[40]], which considers Mandarin characters and English words as the tokens in the edit distance calculation, on three designed test sets and one well-known, publicly available test set (i.e. AIShell-1[[41]] test) with Kaldi, ESPNet and WeNet toolkits respectively.5253## Citation5455```bibtex56@misc{zhang2021wenetspeech,57 title={WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition},58 author={Zhang et al. (2021)},59 year={2021},60 note={arXiv:2110.03370}61}62```6364- arXiv: 2110.03370
qhjqhj00