Se Asr Wer Eval

se-asr-wer-eval

How Bad Are Artifacts?: Analyzing the Impact of Speech Enhancement Errors on ASR — Iwamoto et al. (2022) (arXiv:2201.06685, 2022)

What this evaluates

Evaluates how speech enhancement (SE) artifacts and noise errors affect automatic speech recognition (ASR) performance. It measures Word Error Rate (WER) on enhanced speech signals derived from simulated and real-world reverberant noisy conditions to isolate the impact of artifact components.

Datasets

• Simulated WSJ0+CHiME-3 — total 40000; splits: train (30000), dev (5000), eval (5000)
• CHiME-3 et05_real — total ?; splits: test (-1)

Metrics

• WER [%] (primary) — range: percent
  • Word Error Rate calculated as the minimum number of insertions, deletions, and substitutions of words required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference.
• SDR [dB] — range: other
  • Signal-to-Distortion Ratio measuring the ratio of target signal power to the total distortion power (noise, artifacts, interference).
• SNR [dB] — range: other
  • Signal-to-Noise Ratio measuring the ratio of target speech power to noise power.
• SAR [dB] — range: other
  • Signal-to-Artifact Ratio measuring the ratio of target speech power to artifact power (unrepresentable signals beyond speech and noise).

Input / output format

Input: Reverberant noisy speech waveforms (simulated from WSJ0 speech + CHiME-3 noise, or real CHiME-3 recordings).

Output: ASR text transcripts (for WER calculation) and enhanced speech waveforms (for SDR/SNR/SAR calculation).

Scoring recipe

def evaluate(transcripts, references, enhanced_wav, clean_wav):
    wer = calculate_wer(transcripts, references)
    sdr = compute_sdr(enhanced_wav, clean_wav)
    snr = compute_snr(enhanced_wav, clean_wav)
    sar = compute_sar(enhanced_wav, clean_wav)
    return {'WER': wer, 'SDR': sdr, 'SNR': snr, 'SAR': sar}

Common pitfalls

• Assuming that improving SNR or SDR directly translates to better ASR; the paper demonstrates that artifacts (low SAR) are the primary cause of WER degradation.
• Using the official CHiME-3 test set instead of the custom simulated dataset; the authors explicitly created a custom mix to allow future multi-speaker extensions.
• Neglecting that the ASR system is trained independently of the SE front-end (multi-condition training), which is a key constraint for deployed systems.

Evidence (verbatim from paper)

We created 30,000, 5,000, and 5,000 noisy speech signals for training, development, and evaluation sets, respectively. The input SNR of the training and development sets was randomly selected for each utterance between 0 dB and 10 dB, while the input SNR of the evaluation set was set to 0 dB. The above simulated dataset is used to analyze the relation between the SE metrics (i.e., SDR, SNR, SAR) and the ASR metric (i.e., word error rate (WER)).

Citation

@misc{iwamoto2022artifacts,
  title={How Bad Are Artifacts?: Analyzing the Impact of Speech Enhancement Errors on ASR},
  author={Iwamoto et al. (2022)},
  year={2022},
  note={arXiv:2201.06685}
}

• arXiv: 2201.06685