Daps Noisy Vctk Eval

daps-noisy-vctk-eval

High Fidelity Speech Regeneration with Application to Speech Enhancement — Polyak et al. (2021) (arXiv:2102.00429, 2021)

What this evaluates

Evaluates speech enhancement models by measuring their ability to restore clean speech from noisy or degraded inputs. It probes perceptual quality, acoustic fidelity, and semantic preservation using human listening tests and objective feature-space distances.

Datasets

• DAPS — total ?; splits: test (-1)
• Noisy VCTK — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• MUSHRA (primary) — range: other
  • Mean Opinion Score for Multiple Stimuli with Hidden Reference and Anchor. Human raters score audio quality on a 0-5 scale, with the clean reference labeled as 5.0.
• FDSD — range: other
  • Fréchet Deep Speech Distance. Computes the Fréchet distance between the activation distributions of randomly sampled generated outputs and the target sample, extracted using a DeepSpeech2 ASR model. Lower is better.
• cFDSD — range: other
  • Conditional Fréchet Deep Speech Distance. Similar to FDSD but computed between the generated output and its specific matched clean target using the same DeepSpeech2 ASR model. Lower is better.

Input / output format

Input: Noisy or degraded speech audio waveform, resampled to 24kHz.

Output: Regenerated/enhanced speech audio waveform, resampled to 24kHz.

Scoring recipe

def evaluate(predictions, gold):
    # MUSHRA: Human raters score 0-5 (clean=5.0)
    mushra = human_rate(predictions, gold)
    # Load DeepSpeech2 ASR model
    asr = load_model('DeepSpeech2')
    # Extract features
    feats_gen = asr.encode(predictions)
    feats_target = asr.encode(gold)
    # FDSD: distance between random generated samples and target
    fdsd = frechet_distance(feats_gen, feats_target)
    # cFDSD: distance between generated sample and matched clean target
    cfdsd = frechet_distance(feats_gen, feats_target)
    return {'MUSHRA': mushra, 'FDSD': fdsd, 'cFDSD': cfdsd}

Common pitfalls

• PESQ is explicitly excluded despite being standard in speech enhancement, as it is designed for codecs/channels and shows low correlation with MOS for this task.
• FDSD measures distance between random generated samples and the target, while cFDSD measures distance between the generated sample and its specific matched clean target.
• VCTK uses artificially added noise, making it less challenging than DAPS which uses real-world recordings.

Evidence (verbatim from paper)

For subjective metric, we used MUSHRA – We ask human raters to compare samples created from the same test signal. The clean sample is presented to the user before the processed files and is labeled with a 5.0 score. For objective metrics we used two distances proposed in[25]: (i) Fréchet Deep Speech Distance (FDSD) - a distance measure calculated between the activations of two randomly sampled sets of output signals to the target sample using DeepSpeech2 ASR model; (ii) Conditional Fréchet Deep Speech Distance (cFDSD) - similar to FDSD, but computed between the generated output to its matched clean target.

Citation

@misc{polyak2021highfidelity,
  title={High Fidelity Speech Regeneration with Application to Speech Enhancement},
  author={Polyak et al. (2021)},
  year={2021},
  note={arXiv:2102.00429}
}

• arXiv: 2102.00429