qhjqhj00 librispeech-chime4-asr-eval
Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASR — Hu et al. (2023) (arXiv:2304.04974, 2023)
What this evaluates
Evaluates the robustness of automatic speech recognition (ASR) models under various noise conditions and signal-to-noise ratios (SNRs) using simulated and real-world noisy speech datasets.
Datasets
- LibriSpeech — total ?; splits: train (-1), val (-1), test (4200)
- CHiME-4 — total ?; splits: train (8738), val (3280), test (2640); repo http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/index.html
Metrics
WER(primary) — range: percent- Word Error Rate, calculated as the number of substitutions, deletions, and insertions required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference.
Input / output format
Input: Noisy audio waveform (16 kHz) and corresponding reference transcript.
Output: Predicted text transcript.
Scoring recipe
def compute_wer(predictions, references):
total_errors = 0
total_words = 0
for pred, ref in zip(predictions, references):
total_errors += levenshtein_distance(pred.split(), ref.split())
total_words += len(ref.split())
return (total_errors / total_words) * 100 if total_words > 0 else 0.0
Common pitfalls
- The paper explicitly states no language model (LM) is used during inference for LibriSpeech, whereas many baselines and prior works do use one.
- CHiME-4 evaluation uses only the one-channel track for validation and testing, despite training on all six channels.
- WER is averaged across SNR levels in the tables, which can mask performance degradation at lower SNRs.
Evidence (verbatim from paper)
“Avg” denotes the averaged WER results on all type-A noise conditions, and “Clean” denotes the WER results on clean test set.
Citation
@misc{hu2023wav2code,
title={Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASR},
author={Hu et al. (2023)},
year={2023},
note={arXiv:2304.04974}
}
- arXiv: 2304.04974