Singing Voice Separation Eval

singing-voice-separation-eval

Adversarial Semi-Supervised Audio Source Separation applied to Singing Voice Extraction — Stoller et al. (2017) (arXiv:1711.00048, 2017)

What this evaluates

This benchmark evaluates a model's ability to extract singing vocals from mixed audio tracks. It probes the effectiveness of adversarial semi-supervised learning in separating sources without relying on perfectly paired mixture-source training data.

Datasets

• DSD100 — total ?; splits: train (-1), test (25)
• iKala — total ?; splits: unlabeled (-1), validation (-1), test (-1)
• MedleyDB — total ?; splits: unlabeled (-1), validation (-1), test (-1)
• CCMixter — total ?; splits: unlabeled (-1), validation (-1), test (-1)

Metrics

• SDR (primary) — range: other
  • Track-wise normalised Signal-to-Distortion Ratio (SDR). Measures the ratio of energy in the target source to the total energy of distortion, interference, and artifacts in the estimate.
• SIR — range: other
  • Track-wise normalised Signal-to-Interference Ratio (SIR). Measures the ratio of energy in the target source to the energy of interfering sources in the estimate.
• SAR — range: other
  • Track-wise normalised Signal-to-Artifacts Ratio (SAR). Measures the ratio of energy in the target source to the energy of artifacts introduced by the separation process.

Input / output format

Input: Mono audio downsampled to 8 kHz, converted to a magnitude spectrogram via a 512-point FFT with 50% overlap, and log-normalized. Batched into 64 spectrogram excerpts.

Output: Log-normalised magnitude spectrograms for each source, reconstructed into time-domain signals via inverse STFT using the mixture's phase.

Scoring recipe

def evaluate(predictions, targets):
    sdr_scores, sir_scores, sar_scores = [], [], []
    for pred, tgt in zip(predictions, targets):
        sdr_scores.append(normalised_sdr(pred, tgt))
        sir_scores.append(normalised_sir(pred, tgt))
        sar_scores.append(normalised_sar(pred, tgt))
    return {
        'SDR': sum(sdr_scores) / len(sdr_scores),
        'SIR': sum(sir_scores) / len(sir_scores),
        'SAR': sum(sar_scores) / len(sar_scores)
    }

Common pitfalls

• Using paired mixture-source data for unsupervised training without accounting for dataset bias; the authors explicitly used multi-track stems but treated them as unpaired to isolate the regularization effect.
• Omitting the accompaniment discriminator, which can bias solutions towards favouring vocal quality over accompaniment quality.
• Failing to match the exact 8 kHz mono preprocessing and log-normalization, which significantly alters the spectrogram distributions fed to the model.

Evidence (verbatim from paper)

For evaluation, we calculate the track-wise (normalised) SDR, SIR, and SAR metrics*[25]*, with mono estimates and target signals sampled at 8 KHz. Table2 shows averages over the test set and its subsets containing only tracks from a specific data source.

Citation

@misc{stoller2017adversarial,
  title={Adversarial Semi-Supervised Audio Source Separation applied to Singing Voice Extraction},
  author={Stoller et al. (2017)},
  year={2017},
  note={arXiv:1711.00048}
}

• arXiv: 1711.00048