Wsj0 Speech Separation Eval

wsj0-speech-separation-eval

Deep clustering: Discriminative embeddings for segmentation and separation — Hershey et al. (2015) (arXiv:1508.04306, 2015)

What this evaluates

This evaluation probes a model's ability to perform single-channel speech separation by learning discriminative time-frequency embeddings that group mixture components into distinct speaker clusters. It specifically tests generalization to unseen speakers and scaling to three-speaker mixtures without retraining.

Datasets

• WSJ0-based Speech Mixtures — total ?; splits: train (-1), val (-1), test_closed (-1), test_open (-1)

Metrics

• SDR improvement (dB) (primary) — range: dB
  • Signal-to-Distortion Ratio (SDR) improvement calculated as the difference between the SDR of the separated signal and the SDR of the original mixture, averaged over test utterances. Computed using the bss_eval toolbox.

Input / output format

Input: Log short-time Fourier spectral magnitudes of the mixture speech, computed with a 32 ms window, 8 ms shift, and square root of Hann window. Input is segmented into 100-frame chunks.

Output: Embedding matrix V mapping each time-frequency bin to a speaker cluster dimension. Separated signals are reconstructed by applying time-frequency masks derived from clustering V.

Scoring recipe

def compute_sdr_improvement(mixture, separated, reference):
    sdr_mix = bss_eval_sdr(mixture, reference)
    sdr_sep = bss_eval_sdr(separated, reference)
    return np.mean(sdr_sep - sdr_mix)

Common pitfalls

• Within-segment clustering suffers from permutation ambiguity; the paper reports oracle permutation results (minimizing L2 distance to reference) as an upper bound, which is not achievable in practice without ground truth.
• Evaluation distinguishes between 'closed speaker' (speakers seen during training) and 'open speaker' (unseen speakers); methods often require speaker adaptation to perform on the open set.
• Training masks out time-frequency bins where source magnitude is below -40 dB of the max, which can bias the model if the evaluation protocol does not account for silent regions similarly.

Evidence (verbatim from paper)

For all the experiment, performance was evaluated in terms of averaged signal-to-distortion ratio (SDR) using the bss_eval toolbox [27]. The initial SDR averaged over the mixtures was 0.16 dB for two speaker mixtures and -2.95 dB for three speaker mixtures.

Citation

@misc{hershey2015deepclustering,
  title={Deep clustering: Discriminative embeddings for segmentation and separation},
  author={Hershey et al. (2015)},
  year={2015},
  note={arXiv:1508.04306}
}

• arXiv: 1508.04306