Av Speech Separation Eval

av-speech-separation-eval

Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation — Ephrat et al. (2018) (arXiv:1804.03619, 2018)

What this evaluates

Evaluates a model's ability to separate target speaker speech from audio mixtures (noise or other speakers) using synchronized visual face cues. It probes speaker-independent audio-visual fusion and robustness to varying numbers of speakers and background noise.

Datasets

• AVSpeech — total ?; splits: train (-1), test (-1)
• AudioSet — total ?; splits: train (-1), test (-1)
• CHiME-2 — total ?; splits: test (-1)
• Mandarin — total ?; splits: test (-1)
• TCD-TIMIT — total ?; splits: test (-1)
• CUAVE — total ?; splits: test (-1)

Metrics

• SDR improvement (primary) — range: dB
  • Signal-to-Distortion Ratio improvement computed using the BSS Eval toolbox. Measures the quality of separated speech relative to the reference mixture.
• PESQ — range: [-2, 4.5]
  • Perceptual Evaluation of Speech Quality. A standard objective metric for speech enhancement quality.
• STOI — range: [0, 1]
  • Short-Time Objective Intelligibility. Predicts speech intelligibility based on temporal envelope correlation.

Input / output format

Input: Single-channel audio mixture (speech + noise/other speakers) synchronized with visual face streams (one or more) of the target speaker(s).

Output: Separated speech signal(s) or time-frequency masks for each target speaker.

Scoring recipe

# Compute SDR improvement via BSS Eval
# ref: clean target speech, est: model output, mix: input mixture
sdr_est = bss_eval_sources(ref, est, frame_length=..., hop_length=...)
sdr_mix = bss_eval_sources(ref, mix, frame_length=..., hop_length=...)
sdr_improvement = sdr_est - sdr_mix
return sdr_improvement

Common pitfalls

• Synthetic test sets are generated on-the-fly from AVSpeech and AudioSet with a 90/10 split, meaning there is no fixed public benchmark for direct comparison.
• Real-world video evaluations lack clean reference audio, so they are strictly qualitative and cannot be scored with SDR/PESQ/STOI.
• Prior audio-visual methods are speaker-dependent, while this evaluation uses a speaker-independent model, complicating direct architectural comparisons.

Evidence (verbatim from paper)

Separated speech quality is evaluated using signal-to-distortion ratio (SDR) improvement from the BSS Eval toolbox [Vincent et al., 2006], a commonly used metric for evaluating speech separation quality (see Section A in the Appendix).

Citation

@misc{ephrat2018looking,
  title={Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation},
  author={Ephrat et al. (2018)},
  year={2018},
  note={arXiv:1804.03619}
}

• arXiv: 1804.03619