Dns Challenge Eval

dns-challenge-eval

The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results — Reddy et al. (2020) (arXiv:2005.13981, 2020)

What this evaluates

Evaluates the perceptual quality and intelligibility of deep noise suppression models under real-world, non-stationary noise conditions. It specifically probes whether models generalize from synthetic training data to real-world acoustic environments.

Datasets

• DNS Challenge Dataset — total ?; splits: test (-1); repo https://github.com/microsoft/DNS-Challenge

Metrics

• ITU-T P.808 (primary) — range: other
  • Crowd-sourced subjective Mean Opinion Score derived from the ITU-T P.808 framework, measuring perceived speech quality after enhancement.
• PESQ — range: [0, 1]
  • Perceptual Evaluation of Speech Quality, a standard objective metric for speech enhancement that correlates with human listening tests.
• STOI — range: [0, 1]
  • Short-Time Objective Intelligibility, a metric that predicts speech intelligibility based on temporal envelope correlation.

Input / output format

Input: Paired clean speech and noisy speech recordings (real-world non-stationary noise conditions).

Output: Enhanced speech waveform.

Scoring recipe

def compute_metrics(predictions, gold):
    # predictions and gold are aligned speech waveforms
    pesq_scores = [pesq(gold[i], predictions[i]) for i in range(len(gold))]
    stoi_scores = [stoi(gold[i], predictions[i]) for i in range(len(gold))]
    # Subjective MOS is obtained via crowd-sourced ITU-T P.808 framework
    mos_scores = get_subjective_mos(predictions)
    return {
        'PESQ': np.mean(pesq_scores),
        'STOI': np.mean(stoi_scores),
        'ITU-T P.808': np.mean(mos_scores)
    }

Common pitfalls

• Models trained exclusively on synthetic noise often degrade significantly when evaluated on real-world recordings, making synthetic-only testing misleading.
• Subjective evaluation requires strict adherence to the ITU-T P.808 framework and careful crowd-sourcing calibration to ensure perceptual scores align with human listeners.

Evidence (verbatim from paper)

The INTERSPEECH 2020 Deep Noise Suppression Challenge introduces a large-scale, real-world representative dataset of clean speech and noise recordings derived from Librivox and Audioset/Freesound, with a focus on high-quality, non-stationary noise conditions. It presents an open-source, scalable online subjective evaluation framework based on ITU-T P.808 to enable reliable, crowd-sourced perceptual testing, addressing the gap between synthetic test performance and real-world deployment.

Citation

@misc{reddy2020dnschallenge,
  title={The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results},
  author={Reddy et al. (2020)},
  year={2020},
  note={arXiv:2005.13981}
}

• arXiv: 2005.13981