Speech Enhancement Eval

speech-enhancement-eval

UniPASE: A Generative Model for Universal Speech Enhancement with High Fidelity and Low Hallucinations — Rong et al. (2026) (arXiv:2604.14606, 2026)

What this evaluates

Evaluates speech enhancement models on their ability to restore audio degraded by multiple distortion types (noise, reverberation, packet loss, clipping, bandwidth limitation, codec artifacts) across varying sampling rates, while preserving speaker identity, phonetic content, and perceptual quality.

Datasets

• DNS 2020 test set — total ?; splits: test (-1)
• PLC 2024 validation set — total ?; splits: val (-1)
• VoiceFixer GSR test set — total ?; splits: test (-1)
• URGENT 2025 non-blind test set — total ?; splits: test (-1); repo https://github.com/urgent-challenge/urgent2025_challenge/tree/main/simulation

Metrics

• DNSMOS (primary) — range: [0, 5]
  • Non-intrusive perceptual quality metric based on ITU-T P.835, predicting mean opinion score from audio features.
• PESQ — range: [0, 4.5]
  • Perceptual Evaluation of Speech Quality, an intrusive metric comparing enhanced and reference speech using a psychoacoustic model.
• ESTOI — range: [0, 1]
  • Extended Short-Time Objective Intelligibility, an intrusive metric predicting speech intelligibility based on temporal modulation spectra.
• WER/CER — range: percent
  • Word Error Rate or Character Error Rate, computed by comparing ASR transcriptions of enhanced speech against reference transcripts.
• SBS — range: [0, 1]
  • SpeechBERTScore, computed as cosine similarity between HuBERT/mHuBERT representations of enhanced and reference speech.

Input / output format

Input: Degraded speech audio waveform (16 kHz, 44.1 kHz, or 48 kHz) with various distortions (noise, reverberation, packet loss, clipping, bandwidth limitation, codec artifacts).

Output: Enhanced/restored speech audio waveform.

Scoring recipe

# Compare enhanced prediction against clean reference
dns = dns_mos(pred, sr=16000)
pesq = pesq(gold, pred, sr=16000)
estoi = estoi(gold, pred, sr=16000)
sbs = cosine_similarity(hubert(gold), hubert(pred))
wer = asr_wer(asr_transcribe(pred), asr_transcribe(gold))
return {"DNSMOS": dns, "PESQ": pesq, "ESTOI": estoi, "SBS": sbs, "WER": wer}

Common pitfalls

• Intrusive metrics like PESQ and ESTOI are unreliable for generative models due to sensitivity to imperceptible signal-level differences.
• For the DNS 2020 with-reverb subset, clean references without reverberation must be used to compute metrics, as the model is also expected to perform dereverberation.
• ASR-based metrics (WER/CER) require careful reference selection; when transcripts are missing, pseudo-references from clean speech ASR are used.

Evidence (verbatim from paper)

Following the URGENT 2025 Challenge, we report a comprehensive set of evaluation metrics spanning perceptual quality, intelligibility, speaker similarity, and linguistic correctness: Non-intrusive metrics: DNSMOS (16 kHz), UTMOS (16 kHz), and NISQA (48 kHz)... Intrusive metrics: Perceptual evaluation of speech quality (PESQ) and extended short-time objective intelligibility (ESTOI)... These metrics are sensitive to imperceptible signal-level differences and may be unreliable for generative models.

Citation

@misc{rong2026unipase,
  title={UniPASE: A Generative Model for Universal Speech Enhancement with High Fidelity and Low Hallucinations},
  author={Rong et al. (2026)},
  year={2026},
  note={arXiv:2604.14606}
}

• arXiv: 2604.14606