Chime4 Asr Robustness Eval

chime4-asr-robustness-eval

Speaker Reinforcement Using Target Source Extraction for Robust Automatic Speech Recognition — Zorila et al. (2022) (arXiv:2205.04433, 2022)

What this evaluates

Evaluates the robustness of automatic speech recognition systems in noisy environments by measuring word error rates on enhanced speech from speaker extraction models across matched and mismatched noisy conditions.

Datasets

• CHiME-4 — total ?; splits: test (-1)
• VoiceBank-DEMAND — total ?; splits: test (-1)
• WHAM! — total ?; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: the percentage of words incorrectly recognized (substitutions, deletions, insertions) relative to the total number of words in the reference transcript.
• SDR — range: other
  • Signal-to-Distortion Ratio: a separation quality metric computed using the BSSeval toolkit, reported in decibels (dB).
• STOI — range: [0, 1]
  • Short-Time Objective Intelligibility: a perceptual metric predicting speech intelligibility, computed via a PyTorch implementation.

Input / output format

Input: Single-channel 16kHz noisy speech audio waveforms, along with target speaker enrolment utterances for the extraction model.

Output: ASR system transcriptions (text) for WER calculation; enhanced audio waveforms for SDR/STOI calculation.

Scoring recipe

def compute_metrics(predictions, gold, ref_audio, enh_audio):
    # WER calculation
    wer = 100 * edit_distance(predictions, gold) / len(gold.split())
    # SDR calculation via BSSeval
    sdr = bss_eval_sdr(ref_audio, enh_audio)
    # STOI calculation via PyTorch implementation
    stoi = pytorch_stoi(ref_audio, enh_audio)
    return {'wer': wer, 'sdr': sdr, 'stoi': stoi}

Common pitfalls

• Mismatched test conditions across datasets: CHiME-4 et05 is evaluated under matched conditions, while VoiceBank-DEMAND and WHAM! are evaluated under mismatched conditions.
• Enrolment signal selection differs by dataset type: clean waveforms are used for simulated CHiME-4, VBD, and WHAM!, but close-talk microphone recordings are used for real CHiME-4.
• Two distinct ASR acoustic models are used (clean-trained WSJ vs. noisy-trained CHiME-4), which significantly affects baseline WER and should be reported separately.

Evidence (verbatim from paper)

The performance is mainly assessed in terms of word error rate (WER), however, for some preliminary experiments, the signal-to-distortion (SDR) and STOI values are reported as well. The SDR scores are computed using the BSSeval toolkit [24], and the STOI training loss is computed using a freely available PyTorch implementation2.

Citation

@misc{zorila2022speaker,
  title={Speaker Reinforcement Using Target Source Extraction for Robust Automatic Speech Recognition},
  author={Zorila et al. (2022)},
  year={2022},
  note={arXiv:2205.04433}
}

• arXiv: 2205.04433