Zero Shot Asr Eval

zero-shot-asr-eval

When Denoising Hinders: Revisiting Zero-Shot ASR with SAM-Audio and Whisper — Islam et al. (2026) (arXiv:2603.04710, 2026)

What this evaluates

Evaluates the impact of perceptual audio enhancement (SAM-Audio) on zero-shot automatic speech recognition performance across Bengali and English noisy speech. It probes whether signal-level quality improvements translate to better machine transcription accuracy.

Datasets

• Bengali Noisy YouTube dataset — total ?; splits: test (-1)
• English noisy dataset — total ?; splits: test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate: ratio of edit operations (substitutions, deletions, insertions) to the total number of words in the reference transcription.
• CER (primary) — range: percent
  • Character Error Rate: ratio of character-level edit operations to the total number of characters in the reference transcription.
• BLEU-F1 — range: [0, 1]
  • F1 score combining BLEU precision and recall metrics for n-gram matching.
• RTF — range: other
  • Real-Time Factor: ratio of wall-clock processing time to the duration of the input audio.
• PSNR — range: other
  • Peak Signal-to-Noise Ratio in decibels, computed between clean reference audio and noisy or denoised audio signals.

Input / output format

Input: Raw noisy audio files or SAM-Audio processed (denoised) audio files.

Output: Predicted text transcription of the audio.

Scoring recipe

def compute_asr_metrics(pred, ref):
    wer = edit_distance_words(pred, ref) / len(ref_words)
    cer = edit_distance_chars(pred, ref) / len(ref_chars)
    return wer, cer

def compute_psnr(clean, processed):
    mse = np.mean((clean - processed) ** 2)
    psnr = 10 * np.log10(max_signal_power / mse)
    return psnr

Common pitfalls

• Assuming higher perceptual quality (e.g., higher PSNR) automatically improves ASR performance.
• Averaging metrics without checking utterance-level distribution shifts, which can mask systematic degradation across most clips.
• Evaluating only small Whisper variants, as larger models show more pronounced sensitivity to denoising artifacts.

Evidence (verbatim from paper)

This section reports zero-shot ASR performance of Whisper models on the Bengali Noisy YouTube dataset and an English noisy dataset. We use Word Error Rate (WER) and Character Error Rate (CER) as the primary metrics, while BLEU-F1 and Real-Time Factor (RTF) are included for completeness (Tables I–IV).

Citation

@misc{islam2026whisperdenoising,
  title={When Denoising Hinders: Revisiting Zero-Shot ASR with SAM-Audio and Whisper},
  author={Islam et al. (2026)},
  year={2026},
  note={arXiv:2603.04710}
}

• arXiv: 2603.04710