Moisesdb Eval

moisesdb-eval

Moisesdb: A dataset for source separation beyond 4-stems — Pereira et al. (2023) (arXiv:2307.15913, 2023)

What this evaluates

Evaluates the ability of audio source separation models to isolate individual instrument stems from mixed stereo recordings. It probes fine-grained separation capabilities across a hierarchical taxonomy of up to 11 stems, testing robustness to stem imbalance and un-mastered audio characteristics.

Datasets

• MoisesDB — total 240; splits: test (-1); repo https://github.com/moises-ai/moises-db

Metrics

• SDR (primary) — range: dB
  • Signal-to-Distortion Ratio in decibels. Computed as 10 * log10(||s_true||^2 / ||s_true - s_est||^2), where s_true is the ground truth stem and s_est is the estimated stem. Higher values indicate better separation quality.

Input / output format

Input: Stereo mixed audio recording of a song.

Output: One or more separated audio stems (waveform files) corresponding to the requested instrument classes (e.g., vocals, drums, bass, piano, guitar, etc.).

Scoring recipe

def compute_sdr(gt, est):
    num = np.sum(gt ** 2)
    den = np.sum((gt - est) ** 2)
    return 10 * np.log10(num / den) if den > 0 else float('inf')

def evaluate(dataset, preds):
    scores = []
    for track in dataset:
        for stem in track.stems:
            scores.append(compute_sdr(track[stem], preds[track.id][stem]))
    return np.mean(scores)

Common pitfalls

• The dataset contains un-mastered mixes with lower loudness and higher dynamic range, causing distributional shift when evaluating on commercial mastered tracks.
• Stem and track frequencies follow a power-law distribution, making underrepresented stems (e.g., 'wind', 'contrabass') significantly harder to separate and potentially skewing overall averages.
• Not all songs contain all stems; the number of available tracks varies per stem configuration (e.g., N=235 for 4-stem, N=104 for 5-stem), so metrics must be averaged over available tracks rather than assuming a fixed set.

Evidence (verbatim from paper)

Table 3: Mean, standard deviation (Std), and median (Mdn) of the SDR in dB for each Model/Method and stem type. The varying number of available tracks is denoted by N. Overall indicates performance over all tracks regardless of stem group.

Citation

@misc{pereira2023moisesdb,
  title={Moisesdb: A dataset for source separation beyond 4-stems},
  author={Pereira et al. (2023)},
  year={2023},
  note={arXiv:2307.15913}
}

• arXiv: 2307.15913