Ersb Eval

ersb-eval

Towards General Discrete Speech Codec for Complex Acoustic Environments: A Study of Reconstruction and Downstream Task Consistency — Wang et al. (2025) (arXiv:2505.22515, 2025)

What this evaluates

This benchmark evaluates the environmental resilience of discrete speech codecs by measuring how reconstruction quality and downstream task performance degrade under varying signal-to-noise ratios, loudness levels, and real-world acoustic conditions. It probes both signal fidelity and semantic/intelligibility consistency after codec compression and subsequent speech enhancement or recognition.

Datasets

• Environment-Resilient Speech Codec Benchmark (ERSB) — total ?; splits: simulated (-1), real-world (-1)

Metrics

• PESQ (primary) — range: other
  • Perceptual Evaluation of Speech Quality score computed between reference and reconstructed audio using the VERSA toolkit.
• STOI (primary) — range: [0, 1]
  • Short-Time Objective Intelligibility score measuring speech intelligibility between reference and reconstructed audio.
• SI-SDR — range: dB
  • Scale-Invariant Signal-to-Distortion Ratio in dB, measuring enhancement performance before and after codec reconstruction.
• WER — range: percent
  • Word Error Rate in percent, measuring ASR transcription accuracy on enhanced speech after codec reconstruction.

Input / output format

Input: Clean or noisy speech audio (simulated via DNS5/CHiME1/RIR mixing or real-world recordings from DNS1/CHiME4).

Output: Reconstructed speech audio after codec compression/decompression. Downstream outputs include enhanced audio (via SepFormer) and transcriptions (via Whisper Large-v3).

Scoring recipe

# Reconstruction metrics
pesq = compute_pesq(ref_audio, codec_out)
stoi = compute_stoi(ref_audio, codec_out)

# Downstream consistency metrics
enh_codec = run_se(codec_out)  # SepFormer
enh_ref = run_se(ref_audio)
delta_si_sdr = compute_si_sdr(enh_codec, ref_audio) - compute_si_sdr(enh_ref, ref_audio)

trans_codec = run_asr(enh_codec)  # Whisper
trans_ref = run_asr(enh_ref)
delta_wer = compute_wer(trans_codec, ref_text) - compute_wer(trans_ref, ref_text)

Common pitfalls

• ΔSI-SDR and ΔWER measure relative degradation rather than absolute performance; negative ΔSI-SDR or positive ΔWER indicates codec-induced harm.
• Simulated SNR/loudness variations do not capture real-world noise distributions; real-world tests require TSI-SDR binning via TorchAudio-Squim.
• Codecs are evaluated at fixed bitrates; comparing across different bitrate configurations without normalization can skew results.

Evidence (verbatim from paper)

For the evaluation of signal metrics, we select the VERSA toolkit [22] to measure the PESQ, STOI, and SI-SDR metrics. ... Table 3: ΔTSI-SDR (dB) of SE task and ΔWER (%) of SE+ASR task on the TSI-SDR-divided CHiME4.

Citation

@misc{wang2025towards,
  title={Towards General Discrete Speech Codec for Complex Acoustic Environments: A Study of Reconstruction and Downstream Task Consistency},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.22515}
}

• arXiv: 2505.22515