Cleanunet Eval

cleanunet-eval

Speech Denoising in the Waveform Domain with Self-Attention — Kong et al. (2022) (arXiv:2202.07790, 2022)

What this evaluates

This evaluation probes a model's ability to remove background noise from speech signals directly in the waveform domain. It measures perceptual quality, speech intelligibility, and computational efficiency using standardized objective metrics and crowdsourced subjective listening tests.

Datasets

• DNS dataset — total ?; splits: test (-1)
• Valentini dataset — total ?; splits: test (-1)
• Internal dataset — total ?; splits: test (-1)

Metrics

• PESQ (WB/NB) — range: [0, 5]
  • Perceptual Evaluation of Speech Quality predicts human perception of speech quality. Computed in Wideband (WB) or Narrowband (NB) mode depending on sampling rate.
• STOI — range: [0, 100] percent
  • Short-Time Objective Intelligibility predicts speech intelligibility by correlating temporal envelopes of clean and enhanced speech.
• PESQ-MOS (SIG/BAK/OVRL) (primary) — range: [1, 5]
  • Mean Opinion Score prediction derived from PESQ features, estimating distortion (SIG), background noise intrusiveness (BAK), and overall quality (OVRL).
• RTF — range: other
  • Real-Time Factor measures inference speed, calculated as the time required to generate the speech divided by the total duration of the speech.

Input / output format

Input: Noisy speech audio clips (typically 10 seconds at 16kHz for DNS/Internal, or 1.33–1.5 seconds at 48kHz for Valentini).

Output: Denoised speech audio waveform matching the input duration and sampling rate.

Scoring recipe

def compute_metrics(clean, pred, sr=16000):
    pesq_wb = pesq(sr, clean, pred, 'wb')
    pesq_nb = pesq(sr, clean, pred, 'nb')
    stoi_val = stoi(clean, pred)
    mos_sig, mos_bak, mos_ovrl = pesq_mos_predict(clean, pred)
    return {'PESQ_WB': pesq_wb, 'PESQ_NB': pesq_nb,
            'STOI': stoi_val * 100, 'MOS_SIG': mos_sig,
            'MOS_BAK': mos_bak, 'MOS_OVRL': mos_ovrl}

# Aggregate over test set
test_results = [compute_metrics(gold, model(noisy)) for gold, noisy in test_set]
avg_metrics = {k: np.mean([r[k] for r in test_results]) for k in test_results[0]}

Common pitfalls

• PESQ is computed at different sampling rates (16kHz vs 48kHz) across datasets, but Wideband (WB) vs Narrowband (NB) modes must be matched correctly to the sampling rate to avoid invalid scores.
• The internal dataset is not publicly released, preventing independent reproduction or direct comparison on the third benchmark.
• PESQ-MOS refers to an algorithmic prediction metric, while the subjective MOS test requires actual human listeners on Mechanical Turk; confusing the two leads to incorrect reporting of evaluation methodology.

Evidence (verbatim from paper)

Evaluation: We conduct both objective and subjective evaluations for denoised speech. Objective evaluation methods include (1) Perceptual Evaluation of Speech Quality (PESQ), (2) Short-Time Objective Intelligibility (STOI), and (3) Mean Opinion Score (MOS) prediction of the i) distortion of speech signal (SIG), ii) intrusiveness of background noise (BAK), and iii) overall quality (OVRL). For subjective evaluation, we perform a MOS test as recommended in ITU-T P.835. We launched a crowd source evaluation on Mechanical Turk. We randomly select 100 utterances from the test set, and each utterance is scored by 15 workers along three axis: SIG, BAK, and OVRL.

Citation

@misc{kong2022cleanunet,
  title={Speech Denoising in the Waveform Domain with Self-Attention},
  author={Kong et al. (2022)},
  year={2022},
  note={arXiv:2202.07790}
}

• arXiv: 2202.07790