Robust Asr Wer Eval

robust-asr-wer-eval

An Investigation of End-to-End Models for Robust Speech Recognition — Prasad et al. (2021) (arXiv:2102.06237, 2021)

What this evaluates

Evaluates the robustness of end-to-end automatic speech recognition models against various stationary and non-stationary noise types at different signal-to-noise ratios (SNR). It also measures the degradation of recognition accuracy on clean speech when noise-adaptation techniques are applied.

Datasets

• Custom noisy speech dataset (7 noise types) — total ?; splits: test (-1); repo https://github.com/archiki/Robust-E2E-ASR

Metrics

• WER (primary) — range: percent
  • Word Error Rate, calculated as the minimum number of word edits (insertions, deletions, substitutions) required to change the predicted transcription into the reference transcription, divided by the total number of words in the reference. Reported as a percentage.

Input / output format

Input: Noisy speech audio recordings mixed with one of seven noise types (Babble, Airport/Station, AC/Vacuum, Cafe, Traffic, Metro, Car) at SNR levels of 0, 5, 10, 15, or 20 dB, plus clean speech recordings.

Output: Predicted text transcription of the audio input.

Scoring recipe

def compute_wer(predictions, references):
    total_words = sum(len(ref.split()) for ref in references)
    if total_words == 0: return 0.0
    edits = 0
    for pred, ref in zip(predictions, references):
        edits += levenshtein_distance(pred.split(), ref.split())
    return (edits / total_words) * 100

Common pitfalls

• Ignoring clean speech WER degradation when evaluating noise-adaptation techniques, as some methods (e.g., SE-VCAE, AvT) significantly harm clean speech performance.
• Failing to distinguish between stationary noise types (Car, Metro, Traffic) and non-stationary noise types (Babble, Airport/Station, Cafe, AC/Vacuum), as adaptation strategies perform differently on each.
• Assuming higher objective speech enhancement scores directly translate to lower ASR WER, as content distortion in enhanced speech can negatively impact recognition.

Evidence (verbatim from paper)

Table 2 lists an exhaustive comparison of all previously mentioned techniques on seven different noise types and five different SNR values. We also report the WER on clean speech to observe degradation with each noise adaptation technique in place.

Citation

@misc{prasad2021robust,
  title={An Investigation of End-to-End Models for Robust Speech Recognition},
  author={Prasad et al. (2021)},
  year={2021},
  note={arXiv:2102.06237}
}

• arXiv: 2102.06237