Rats Asr Wer Eval

rats-asr-wer-eval

Noise-robust Speech Recognition with 10 Minutes Unparalleled In-domain Data — Chen Chen et al. (2022) (arXiv:2203.15321, 2022)

What this evaluates

This evaluation probes the robustness of automatic speech recognition (ASR) systems when trained on extremely limited in-domain noisy data. It measures how well a model can generalize to real-world noisy conditions by leveraging synthetic noisy data generated via a GAN, compared to traditional data augmentation and fine-tuning baselines.

Datasets

• RATS (Channel A) — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• WER (%) (primary) — range: percent
  • Word Error Rate calculated as (Substitutions + Deletions + Insertions) / Total Reference Words, expressed as a percentage. Lower values indicate better transcription accuracy.

Input / output format

Input: 80-dimensional log-mel spectrogram features extracted from audio utterances.

Output: Sequence of byte-pair-encoding (BPE) tokens from a vocabulary of size 994.

Scoring recipe

def calculate_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        # Align using Levenshtein distance
        dist = levenshtein_distance(pred, ref)
        total_errors += dist
        total_words += len(ref)
    wer = (total_errors / total_words) * 100 if total_words > 0 else 0.0
    return wer

Common pitfalls

• Failing to apply speed perturbation (×0.9, ×1.0, ×1.1) during training, which the paper notes significantly lowers WER across all baselines.
• Misinterpreting data requirements: the proposed method uses 10 minutes of unlabeled noisy data for GAN training, whereas baselines like Finetune use labeled data and Mixup uses unlabeled noisy data from the full 44.3-hour set.
• Comparing single-path and dual-path ASR results without noting that the dual-path system inherently yields lower WER due to its architecture, not just data generation.

Evidence (verbatim from paper)

Table 3 summarizes the comparison between the proposed Simu-GAN and other competitive techniques in terms of the WER (%). We observe that the proposed Simu-GAN obtained the best performance. Comparing with the “Mixup” methods, the proposed Simu-GAN achieves the 7.3% absolute WER improvements.

Citation

@misc{chen2022noiserobust,
  title={Noise-robust Speech Recognition with 10 Minutes Unparalleled In-domain Data},
  author={Chen Chen et al. (2022)},
  year={2022},
  note={arXiv:2203.15321}
}

• arXiv: 2203.15321