Opensep Eval

opensep-eval

OpenSep: Leveraging Large Language Models with Textual Inversion for Open World Audio Separation — Tanvir Mahmud, Diana Marculescu (2024) (arXiv:2409.19270, 2024)

What this evaluates

Evaluates open-world audio source separation by measuring how well a model disentangles multiple audio sources from a mixed input. It probes the model's ability to generalize to seen and unseen audio classes and handle complex natural mixtures without manual intervention.

Datasets

• MUSIC — total ?; splits: test (-1)
• VGGSound — total 162433; splits: test (-1)
• AudioCaps — total 44309; splits: test (-1)

Metrics

• SDR (primary) — range: other
  • Signal-to-Distortion Ratio; measures overall separation quality of the target sound relative to the mixture. Computed in dB after aligning predictions to ground truth via best-match permutation.
• SIR — range: other
  • Signal-to-Interference Ratio; estimates the amount of interference from other sources in the separated output. Computed in dB after permutation alignment.

Input / output format

Input: Noisy audio mixture (synthetic or natural) containing 2-6 sources.

Output: Separated audio tracks, one per source in the mixture.

Scoring recipe

def compute_metrics(predictions, ground_truth):
    aligned_preds = find_best_permutation(predictions, ground_truth)
    sdr_scores, sir_scores = [], []
    for pred, gt in zip(aligned_preds, ground_truth):
        sdr_scores.append(compute_sdr(pred, gt))
        sir_scores.append(compute_sir(pred, gt))
    return mean(sdr_scores), mean(sir_scores)

Common pitfalls

• Failing to apply permutation-invariant matching (best-match alignment) between predicted sources and ground truth before computing SDR/SIR.
• Evaluating on natural mixtures (AudioCaps) using automatic metrics instead of human preference studies, as ground truth sources are unavailable.
• Confusing the 'seen' vs 'unseen' class splits, where unseen evaluation strictly uses 50% held-out classes for testing.

Evidence (verbatim from paper)

We use signal-to-distortion ratio (SDR) and signal-to-interference ration (SIR) (Vincent et al., 2006) for evaluating different models. In general, SDR estimates overall separation quality of the target sound, which is widely used in prior work (Dong et al., 2022; Mahmud et al., 2024). SIR estimates the amount of interference in separation from other sources present in the mixture.

Citation

@misc{mahmud2024opensep,
  title={OpenSep: Leveraging Large Language Models with Textual Inversion for Open World Audio Separation},
  author={Tanvir Mahmud, Diana Marculescu (2024)},
  year={2024},
  note={arXiv:2409.19270}
}

• arXiv: 2409.19270