Podcastmix Eval

podcastmix-eval

PodcastMix: A dataset for separating music and speech in podcasts — Schmidt et al. (2022) (arXiv:2207.07403, 2022)

What this evaluates

Evaluates the quality of monaural music and speech source separation in podcast audio. It measures both objective signal fidelity using BSS-eval metrics and subjective perceptual quality using standardized listening tests.

Datasets

• PodcastMix — total ?; splits: synth test (-1), real with-reference (-1), real no-reference (-1); repo https://github.com/MTG/Podcastmix

Metrics

• SDR (primary) — range: dB
  • Source-to-distortion ratio measuring the ratio of target source energy to distortion energy in decibels.
• SI-SDR — range: dB
  • Scale-invariant SDR, computed after optimal scaling of the estimate to match the reference, robust to amplitude changes.
• OVRL — range: [1, 5]
  • Overall quality mean opinion score from webMUSHRA subjective tests, rated on a 1-5 scale.
• SIG — range: [1, 5]
  • Speech quality/distortion mean opinion score from webMUSHRA, rated on a 1-5 scale.
• BAK — range: [1, 5]
  • Background intrusiveness mean opinion score from webMUSHRA, rated on a 1-5 scale.

Input / output format

Input: 44.1kHz mono audio waveform of a podcast mixture containing speech and music.

Output: Two separated audio waveforms: one for speech and one for music.

Scoring recipe

def compute_metrics(reference, estimate, subjective_ratings=None):
    sdr = bss_eval_sdr(reference, estimate)
    si_sdr = compute_si_sdr(reference, estimate)
    if subjective_ratings:
        ovrl = mean(r.ovrl for r in subjective_ratings)
        sig = mean(r.sig for r in subjective_ratings)
        bak = mean(r.bak for r in subjective_ratings)
    return {'SDR': sdr, 'SI-SDR': si_sdr, 'OVRL': ovrl, 'SIG': sig, 'BAK': bak}

Common pitfalls

• BSS-eval metrics (SDR, SIR, SAR) require reference stems, so they cannot be computed for the 'PodcastMix-real no-reference' split.
• Models trained on synthetic data often show severe generalization gaps on real podcast audio, leading to artificially low objective and subjective scores.
• Subjective testing splits listeners: naive listeners only rate OVRL, while experienced listeners rate SIG and BAK, making direct comparison across all metrics require careful pooling.

Evidence (verbatim from paper)

We primarily rely on BSS_eval metrics: source-to-distortion ratio (SDR), source-to-interference ratio (SIR), and source-to-artifact ratio (SAR). Further, as recommended by previous works, we also report scale-invariant SDR (SI-SDR). Since the above metrics require reference stems, we report those for PodcastMix-synth test and PodcastMix-real with-reference sets.

Citation

@misc{schmidt2022podcastmix,
  title={PodcastMix: A dataset for separating music and speech in podcasts},
  author={Schmidt et al. (2022)},
  year={2022},
  note={arXiv:2207.07403}
}

• arXiv: 2207.07403