Robust Asr Eval

robust-asr-eval

Towards Decoupling Frontend Enhancement and Backend Recognition in Monaural Robust ASR — Yang et al. (2024) (arXiv:2403.06387, 2024)

What this evaluates

Evaluates the robustness of monaural automatic speech recognition systems under noisy and reverberant conditions. It measures how well a decoupled frontend speech enhancement module improves the word error rate of a backend ASR model trained exclusively on clean speech.

Datasets

• WSJ0 SI-84 — total 8674; splits: train (7138), val (1206), test (330)
• CHiME-2 — total ?; splits: train (7138), val (409), test (1980)
• LibriSpeech — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate calculated as the percentage of incorrectly recognized words (substitutions, deletions, insertions) relative to the reference transcript length.
• STOI — range: [0, 1]
  • Short-Time Objective Intelligibility, a correlation-based metric that predicts speech intelligibility on a scale from 0 to 1.
• PESQ — range: [-0.5, 4.5]
  • Perceptual Evaluation of Speech Quality, a standardized ITU-T metric for assessing speech quality, typically ranging from -0.5 to 4.5.

Input / output format

Input: 16 kHz monaural speech waveform. For SE, noisy/reverberant speech frames (16 ms window, 2 ms shift, RMS normalized). For ASR, log-Mel spectrograms derived from STFT of enhanced or raw speech.

Output: For SE: enhanced speech waveform. For ASR: recognized text transcript.

Scoring recipe

def compute_metrics(clean_ref, enhanced_pred, asr_hypothesis, asr_reference):
    stoi = compute_stoi(clean_ref, enhanced_pred)
    pesq = compute_pesq(clean_ref, enhanced_pred, fs=16000)
    wer = (edit_distance(asr_hypothesis, asr_reference) / len(asr_reference.split())) * 100
    return {'STOI': stoi, 'PESQ': pesq, 'WER': wer}

Common pitfalls

• Frontend and backend are trained independently; joint training is explicitly avoided to prevent distortion-induced mismatch.
• Reverberation test conditions use direct-path speech alignment to ensure fair WER comparison across different RIRs.
• Noise and reverberation are generated dynamically during training, so models are evaluated on unseen RIRs and noise types.

Evidence (verbatim from paper)

The SE performance is evaluated using standard STOI and perceptual evaluation of speech quality (PESQ) metrics. STOI ranges typically between [0, 1] and indicates speech intelligibility, usually in percentage. PESQ ranges between [-0.5, 4.5] and a higher score denotes higher speech quality. TABLE I: ASR (%WER) Results of the Proposed DN-ARN System and Comparison Systems on WSJ.

Citation

@misc{yang2024decoupling,
  title={Towards Decoupling Frontend Enhancement and Backend Recognition in Monaural Robust ASR},
  author={Yang et al. (2024)},
  year={2024},
  note={arXiv:2403.06387}
}

• arXiv: 2403.06387