Rats Channel A Wer Eval

rats-channel-a-wer-eval

Interactive Feature Fusion for End-to-End Noise-Robust Speech Recognition — Hu et al. (2021) (arXiv:2110.05267, 2021)

What this evaluates

This evaluation probes the robustness of end-to-end automatic speech recognition systems in noisy acoustic conditions. It measures how well a model preserves speech intelligibility and correctly transcribes utterances when background noise is present, specifically testing the mitigation of over-suppression artifacts during joint speech enhancement and recognition.

Datasets

• RATS Channel-A — total ?; splits: train (-1), valid (-1), test (-1); repo https://github.com/YUCHEN005/RATS-Channel-A-Speech-Data

Metrics

• WER(%) (primary) — range: percent
  • Word Error Rate calculated as the minimum number of insertions, deletions, and substitutions required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference. Expressed as a percentage.

Input / output format

Input: Noisy speech waveforms (segmented into several-second chunks) or corresponding Fbank features.

Output: Sequence of 994 BPE tokens representing the transcribed speech.

Scoring recipe

def calculate_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        # Standard edit distance for ASR
        dist = levenshtein_distance(pred.split(), ref.split())
        total_errors += dist
        total_words += len(ref.split())
    wer = (total_errors / total_words) * 100 if total_words > 0 else 0.0
    return wer

Common pitfalls

• Evaluating on the full RATS corpus instead of the specified Channel-A subset, which uses ultra-high frequency recordings and has different noise characteristics.
• Comparing systems trained with different SE/ASR architectures or multi-task loss weights, which invalidates the WER(%) comparison due to architectural confounds.
• Ignoring the over-suppression phenomenon where aggressive noise removal removes speech content, leading to artificially low WER on clean segments but poor generalization on noisy test data.

Evidence (verbatim from paper)

Table 3 summarizes the comparison between the proposed IFF-Net and other competitive methods. Specifically, E2E ASR System denotes the best result of the Conformer-based ASR system, which indicates the difficulty of noise-robust ASR. Cascaded SE and ASR System could slightly improve the performance with help of the enhanced speech. Joint Training Approach achieves 2.5% absolute improvement over E2E ASR System by jointly optimizing the SE module and ASR module. GRF Network can further lower the WER by combining the information of enhanced speech and noisy speech. We observe that the proposed IFF-Net obtains the best result with 4.1% absolute WER reduction over the best baseline.

Citation

@misc{hu2021interactive,
  title={Interactive Feature Fusion for End-to-End Noise-Robust Speech Recognition},
  author={Hu et al. (2021)},
  year={2021},
  note={arXiv:2110.05267}
}

• arXiv: 2110.05267