Sound Demixing Challenge 2023 Eval

sound-demixing-challenge-2023-eval

The Sound Demixing Challenge 2023 $\unicode{x2013}$ Music Demixing Track — Fabbro et al. (2023) (arXiv:2308.06979, 2023)

What this evaluates

Evaluates music source separation models on their ability to isolate individual instruments (vocals, bass, drums, other) from mixed audio tracks. It specifically probes robustness to label noise and bleeding artifacts in training data, as well as standard separation performance across different leaderboards.

Datasets

• SDXDB23_LabelNoise — total ?; splits: test (-1)
• SDXDB23_Bleeding — total ?; splits: test (-1)
• Standard (MDXDB21) — total ?; splits: test (-1)

Metrics

• SDR (Signal-to-Distortion Ratio) (primary) — range: dB
  • Computed in decibels (dB) as 10 * log10(||target||^2 / ||prediction - target||^2). The paper reports Global SDR (mean across all instruments) and per-instrument SDR for Bass, Drums, Other, and Vocals.

Input / output format

Input: Mixed audio waveform of a song containing multiple instruments.

Output: Four separated audio waveforms corresponding to the target instruments: Bass, Drums, Other, and Vocals.

Scoring recipe

def compute_sdr(pred, gt):
    target_energy = np.sum(gt ** 2)
    error_energy = np.sum((pred - gt) ** 2)
    if error_energy == 0:
        return float('inf')
    return 10 * np.log10(target_energy / error_energy)

final_score = np.mean([compute_sdr(pred_bass, gt_bass),
                       compute_sdr(pred_drums, gt_drums),
                       compute_sdr(pred_other, gt_other),
                       compute_sdr(pred_vocals, gt_vocals)])

Common pitfalls

• SDR is reported in decibels (dB), not as a linear ratio; higher values indicate better separation quality.
• The challenge uses different leaderboards for different noise conditions (LabelNoise, Bleeding, Standard), so results are not directly comparable across leaderboards without noting the training data corruption type.
• Ensemble methods combine multiple models per instrument, meaning the evaluation must account for per-source weighting rather than a single monolithic model output.

Evidence (verbatim from paper)

Table 1: Final LabelNoise leaderboard (models trained only on SDXDB23_LabelNoise; top-5) ... Global SDR (dB) ... We report in Table 4 the performance of our iterative refinement baseline. First of all, we highlight the impact that the errors in the data have on the performance of the model: training on SDXDB23_LabelNoise degrades the average separation quality by 1.42dB, while training on SDXDB23_Bleeding degrades it by 0.83dB.

Citation

@misc{fabbro2023sounddemixing,
  title={The Sound Demixing Challenge 2023 $\unicode{x2013}$ Music Demixing Track},
  author={Fabbro et al. (2023)},
  year={2023},
  note={arXiv:2308.06979}
}

• arXiv: 2308.06979