Discongan Se Eval

discongan-se-eval

Leveraging Discriminative Latent Representations for Conditioning GAN-Based Speech Enhancement — Shrishti Saha Shetu, E.A.P. Habets, Andreas Brendel (2025) (arXiv:2508.20859, 2025)

What this evaluates

Evaluates speech enhancement models in extremely low SNR conditions by measuring noise suppression, speech quality preservation, and intelligibility using both objective metrics and subjective listening tests.

Datasets

• Low-SNR Dataset — total 1200; splits: test (1200); repo https://github.com/fhgainr/Low-SNR-Eval-Data
• VB-DMD Dataset — total ?; splits: test (-1)
• DNS Non-Reverb Test Dataset — total ?; splits: test (-1)
• DNS Real Recordings — total 300; splits: test (300)

Metrics

• PESQ (primary) — range: [1, 4.5]
  • Perceptual Evaluation of Speech Quality; scores range from 1 (poor) to 4.5 (excellent) based on psychoacoustic modeling of human hearing.
• SI-SDR — range: dB
  • Scale-Invariant Signal-to-Distortion Ratio; measures overall quality independent of amplitude scaling, calculated as the ratio of target signal energy to distortion energy.
• FwSegSNR — range: dB
  • Frequency-weighted Segmental SNR; measures segmental SNR with perceptual weighting in the frequency domain.
• WER/CER — range: percent
  • Word Error Rate and Character Error Rate computed using Whisper ASR on enhanced vs. clean speech signals.
• WebMUSHRA MOS — range: [0, 100]
  • Subjective overall quality rating on a 0–100 scale via multi-stimuli listening test with 14 participants.

Input / output format

Input: Noisy speech audio segments (typically 10 seconds long) sampled at 16 kHz.

Output: Enhanced speech audio (16 kHz), evaluated against clean references or via non-intrusive predictors.

Scoring recipe

def evaluate(pred, ref):
    pesq_val = pesq(ref, pred, fs=16000)
    si_sdr_val = scale_invariant_sdr(ref, pred)
    fwseg_val = freq_weighted_seg_snr(ref, pred)
    wer_val = whisper_wer(ref, pred)
    return {'PESQ': pesq_val, 'SI-SDR': si_sdr_val, 'FwSegSNR': fwseg_val, 'WER': wer_val}

Common pitfalls

• Real recordings (DNS Real) lack clean references, so intrusive metrics like PESQ and SI-SDR cannot be computed; only non-intrusive metrics apply.
• WER/CER are computed using Whisper ASR on clean references rather than ground-truth transcripts, which may introduce transcription bias.
• The low-SNR test set uses non-uniform SNR intervals (e.g., [-15,-12], [-11,-8] dB), requiring careful grouping when reporting results.

Evidence (verbatim from paper)

To evaluate the proposed method and compare it with baseline methods, we used widely adopted intrusive and non-intrusive objective metrics. When clean references were available, we used several intrusive (full-reference) metrics: The perceptual evaluation of speech quality (PESQ) provides scores from 1 (poor) to 4.5 (excellent), including narrowband and wideband variants. The frequency-weighted segmental SNR (FwSegSNR) measures segmental SNR with perceptual weighting in the frequency domain. The scale-invariant signal-to-distortion ratio (SI-SDR) calculates overall quality independent of the amplitude scaling.

Citation

@misc{shetu2025discongan,
  title={Leveraging Discriminative Latent Representations for Conditioning GAN-Based Speech Enhancement},
  author={Shrishti Saha Shetu, E.A.P. Habets, Andreas Brendel (2025)},
  year={2025},
  note={arXiv:2508.20859}
}

• arXiv: 2508.20859