Chime2 Robust Asr Eval

chime2-robust-asr-eval

Time-Domain Speech Enhancement for Robust Automatic Speech Recognition — Yang et al. (2022) (arXiv:2210.13318, 2022)

What this evaluates

Evaluates the effectiveness of a time-domain speech enhancement frontend in improving automatic speech recognition performance on noisy and reverberant speech. It probes the model's ability to enhance speech without introducing distortion that degrades downstream ASR accuracy.

Datasets

• CHiME-2 — total ?; splits: train (7138), val (409), test (1980)

Metrics

• WER (primary) — range: percent
  • Word Error Rate calculated as the minimum number of insertions, deletions, and substitutions of words required to transform the reference transcription into the predicted transcription, divided by the total number of words in the reference.
• STOI — range: [0, 1]
  • Short-Time Objective Intelligibility, a perceptual metric that predicts speech intelligibility based on temporal envelope correlation in octave bands.

Input / output format

Input: Single-channel noisy/reverberant speech waveform sampled at 16 kHz. For the ASR backend, log-Mel features are extracted from the enhanced speech using a Hamming window and STFT.

Output: Word transcription sequence decoded from senone predictions using a CMU pronunciation dictionary and a 5k tri-gram language model.

Scoring recipe

def calculate_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        dist = levenshtein_distance(pred.split(), ref.split())
        total_errors += dist
        total_words += len(ref.split())
    return (total_errors / total_words) * 100

Common pitfalls

• Using anechoic WSJ clean speech as the training target for enhancement instead of reverberant-clean speech, which causes a mismatch due to altered alignments from binaural room impulse responses.
• Assuming higher enhancement quality based on raw SNR or PESQ directly translates to better ASR performance; the paper demonstrates that STOI-based model selection yields superior ASR results.

Evidence (verbatim from paper)

Because STOI is shown to relate to WER [28], thus we also use validation STOI as a model selection criterion in addition to validation PCM loss.

Citation

@misc{yang2022timespeechenhancement,
  title={Time-Domain Speech Enhancement for Robust Automatic Speech Recognition},
  author={Yang et al. (2022)},
  year={2022},
  note={arXiv:2210.13318}
}

• arXiv: 2210.13318