Speech Separation Eval

speech-separation-eval

SonicSim: A customizable simulation platform for speech processing in moving sound source scenarios — Li et al. (2024) (arXiv:2410.01481, 2024)

What this evaluates

Evaluates speech separation models' ability to isolate individual speaker signals from multi-speaker mixtures under various acoustic conditions, including moving sources, environmental noise, and musical noise. It measures both objective signal quality and subjective perceptual metrics to assess generalization from synthetic to real-world dynamic scenarios.

Datasets

• SonicSet — total ?; splits: test (-1)
• RealSEP — total ?; splits: test (-1)
• HumanSEP — total ?; splits: test (-1)
• LRS2-2Mix — total ?; splits: test (-1)
• Libri2Mix — total ?; splits: test (-1)

Metrics

• SI-SNR (primary) — range: dB
  • Signal-to-Interference Ratio in dB, measuring the ratio of target signal power to interference power after optimal scaling.
• SDR — range: dB
  • Signal-to-Distortion Ratio in dB, evaluating overall separation quality including noise and artifacts.
• NB-PESQ — range: other
  • Narrowband Perceptual Evaluation of Speech Quality, a psychoacoustic model predicting speech quality on a scale up to 4.5.
• WB-PESQ — range: other
  • Wideband Perceptual Evaluation of Speech Quality, similar to NB-PESQ but for wider frequency ranges.
• STOI — range: [0, 1]
  • Short-Time Objective Intelligibility, a metric predicting speech intelligibility on a scale from 0 to 1.
• MOS Overall — range: other
  • Mean Opinion Score from subjective listening tests, rating overall audio quality on a scale from 1 to 5.
• NISQA — range: [0, 1]
  • Non-Intrusive Speech Quality and Intelligibility Assessment, a deep learning-based metric predicting quality on a scale from 0 to 1.
• WER — range: percent
  • Word Error Rate, calculated by transcribing separated speech and comparing to ground truth text.

Input / output format

Input: Mixed audio waveform containing two speakers at 16kHz, optionally with environmental or musical noise.

Output: Separated audio waveforms for each individual speaker.

Scoring recipe

def evaluate_separation(predictions, references):
    metrics = {}
    for pred, ref in zip(predictions, references):
        metrics['SI-SNR'] += si_snr(pred, ref)
        metrics['SDR'] += sdr(pred, ref)
        metrics['STOI'] += stoi(pred, ref)
        metrics['PESQ'] += pesq(pred, ref)
        metrics['WER'] += wer_asr(pred, ref)  # via ASR
        metrics['NISQA'] += nisqa_score(pred)
        metrics['MOS'] += mos_score(pred)     # via subjective or predictor
    return {k: v / len(predictions) for k, v in metrics.items()}

Common pitfalls

• Models are evaluated on real-world moving source datasets (RealSEP, HumanSEP) which introduce dynamic acoustic conditions and Doppler effects not present in static synthetic benchmarks.
• Inference uses a 6-second window with 3-second sliding overlap for SonicSet, meaning metrics are computed on overlapping segments rather than independent clips, potentially inflating scores.
• Multiple subjective metrics (MOS, NISQA) are reported alongside objective ones, requiring different evaluation pipelines and potentially conflicting model rankings.

Evidence (verbatim from paper)

We employed a series of metrics to assess the performance of the speech separation benchmark models comprehensively. These metrics cover multiple dimensions of audio quality, including signal quality (SI-SNR (Le Roux et al., 2019) and SDR (Vincent et al., 2006)), speech intelligibility (STOI (Taal et al., 2011) and WER), and subjective quality perception (PESQ (Rix et al., 2001), NISQA (Mittag et al., 2021) and SigMOS (Ristea et al., 2024)).

Citation

@misc{li2024sonicsim,
  title={SonicSim: A customizable simulation platform for speech processing in moving sound source scenarios},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2410.01481}
}

• arXiv: 2410.01481