qhjqhj00 chime4-adapter-eval
Exploration of Adapter for Noise Robust Automatic Speech Recognition — Hao Shi et al. (2024) (arXiv:2402.18275, 2024)
What this evaluates
Evaluates the robustness of Automatic Speech Recognition (ASR) models under various real-world noise conditions (bus, cafe, pedestrian, street junction) using adapter-based fine-tuning and speech enhancement front-ends.
Datasets
- CHiME-4 — total ?; splits: train (-1), dev (-1), test (-1); repo https://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/index.html
Metrics
WER(primary) — range: percent- Word Error Rate (WER) = (Substitutions + Deletions + Insertions) / Total Reference Words × 100%. Standard ASR evaluation metric reported as a percentage.
Input / output format
Input: 16 kHz audio recordings processed into 80-dimension LFMB features, with corresponding ground-truth transcripts.
Output: Predicted text transcription.
Scoring recipe
def compute_wer(predictions, references):
total_words = sum(len(ref.split()) for ref in references)
errors = 0
for pred, ref in zip(predictions, references):
errors += edit_distance(pred.split(), ref.split())
return (errors / total_words) * 100 if total_words > 0 else 0.0
Common pitfalls
- Confusing the development set (channels 1-6) with the evaluation set (channel 5) for final reporting.
- Using simulated noisy data for the final metric calculation instead of the real noisy test sets.
- Failing to average WER across the four distinct noise conditions (bus, str, ped, caf) before reporting the overall score.
Evidence (verbatim from paper)
Table 1: The performance of baseline pretrained ASR model. ... The Channel 5 real noisy data from development and evaluation sets were used for testing.
Citation
@misc{shi2024explorationadapter,
title={Exploration of Adapter for Noise Robust Automatic Speech Recognition},
author={Hao Shi et al. (2024)},
year={2024},
note={arXiv:2402.18275}
}
- arXiv: 2402.18275