Slakh2100 Sep Eval

slakh2100-sep-eval

Multi-Source Diffusion Models for Simultaneous Music Generation and Separation — Mariani et al. (2023) (arXiv:2302.02257, 2023)

What this evaluates

Evaluates the ability of generative models to separate individual musical instrument stems from a mixed audio track. It probes how well the model captures inter-source dependencies and reconstructs clean waveforms for Bass, Drums, Guitar, and Piano.

Datasets

• Slakh2100 — total 2100; splits: train (1500), val (375), test (225)

Metrics

• SI-SDR_i (primary) — range: other
  • Scale-Invariant Signal-to-Distortion Ratio (improved). It measures the ratio between the target signal and the residual error, normalized by the target's energy to be scale-invariant. Formula: 10 log10(||s_hat^T s||^2 / ||s_hat^T s - s||^2) where s is the reference and s_hat is the estimate.

Input / output format

Input: Mixed audio waveform downsampled to 22kHz, processed in ~12-second context windows.

Output: Separated waveform stems for each target instrument (Bass, Drums, Guitar, Piano).

Scoring recipe

def si_sdr_i(reference, estimate):
    alpha = np.dot(estimate, reference) / np.dot(reference, reference)
    target = alpha * reference
    noise = estimate - target
    sdr = 10 * np.log10(np.sum(target**2) / np.sum(noise**2))
    return sdr

Common pitfalls

• Evaluating only on the four most abundant instruments (Bass, Drums, Guitar, Piano) rather than all 31 classes, which may overestimate generalization to unseen instruments.
• Ignoring the impact of the stochasticity parameter (S_churn) and correction steps (R), which significantly alter sample quality and final SI-SDR_i scores.
• Using context windows of ~12 seconds without specifying boundary handling or padding, which can artificially inflate or deflate separation quality at track edges.

Evidence (verbatim from paper)

We use the $ ext{SI-SDR}_{ ext{i}}$ as our evaluation metric (dB – higher is better).

Citation

@misc{mariani2023multisource,
  title={Multi-Source Diffusion Models for Simultaneous Music Generation and Separation},
  author={Mariani et al. (2023)},
  year={2023},
  note={arXiv:2302.02257}
}

• arXiv: 2302.02257