Audio Visual Separation Eval

audio-visual-separation-eval

Co-Separating Sounds of Visual Objects — Gao et al. (2019) (arXiv:1904.07750, 2019)

What this evaluates

This protocol evaluates an audio-visual model's ability to isolate and separate target instrument sounds from mixed multi-source video audio using visual object cues. It quantifies separation accuracy and artifact suppression across held-out test clips and synthetically mixed pairs. The evaluation also probes generalization to unseen object combinations and visually-guided denoising on real-world videos.

Datasets

• MUSIC — total 9456; splits: train (8928), val (259), test (269)
• AudioSet-Unlabeled — total 113756; splits: train (113756)
• AudioSet-SingleSource — total 15; splits: test (15)
• AV-Bench — total 3; splits: test (3)

Metrics

• SDR (primary) — range: dB
  • Signal-to-Distortion Ratio (SDR) measures the overall quality of the separated signal relative to the reference, capturing both interference and artifacts. Reported in decibels (dB).
• SIR — range: dB
  • Signal-to-Interference Ratio (SIR) quantifies how well interfering sources are suppressed relative to the target signal. Reported in decibels (dB).
• SAR — range: dB
  • Signal-to-Artifact Ratio (SAR) measures the absence of artifacts introduced by the separation process, independent of separation accuracy. Reported in decibels (dB).

Input / output format

Input: 6-second audio clips (sub-sampled at 11kHz) converted to 256x256 log-frequency magnitude spectrograms, paired with visual object crops (224x224) or scene images.

Output: Separated magnitude spectrograms (or time-domain audio waveforms) corresponding to each target visual object/source.

Scoring recipe

import mir_eval
import numpy as np

def compute_metrics(pred_audio, ref_audio):
    # mir_eval expects 2D arrays of shape (n_sources, n_samples)
    ref = np.array([ref_audio])
    pred = np.array([pred_audio])
    sdr, sir, sar, _ = mir_eval.separation.bss_eval_sources(
        reference=ref, estimated=pred, compute_permutation=False
    )
    return sdr[0, 0], sir[0, 0], sar[0, 0]

# Average across all test clips to report final metric values.

Common pitfalls

• SAR measures artifacts, not separation accuracy; naively copying the input twice yields high SAR (~80 dB), so relying solely on SAR is misleading.
• Test evaluation relies on synthetically mixing two single-source videos to guarantee ground truth, which differs from real multi-source audio complexity.
• For ungrounded baselines like NMF-MFCC, both possible source-to-target matchings must be evaluated and the best score reported, artificially inflating results.

Evidence (verbatim from paper)

We use the widely used mir eval library to evaluate the source separation and report the standard metrics: Signal-to-Distortion Ration (SDR), Signal-to-Interference Ratio (SIR), and Signal-to-Artifact Ratio (SAR).

Citation

@misc{gao2019coseparating,
  title={Co-Separating Sounds of Visual Objects},
  author={Gao et al. (2019)},
  year={2019},
  note={arXiv:1904.07750}
}

• arXiv: 1904.07750