Vggsound Sep Eval

vggsound-sep-eval

Separate Anything You Describe — Liu et al. (2023) (arXiv:2308.05037, 2023)

What this evaluates

Evaluates zero-shot language-queried audio source separation on human actions, sound-emitting objects, and human-object interactions. The benchmark tests isolation of a target sound from a mixed audio mixture using text labels.

Datasets

• VGGSound — total 1000; splits: test (1000)

Metrics

• SDRi (primary) — range: dB
  • Signal-to-distortion ratio improvement, calculated as the difference between the SDR of the separated output and the original mixture.
• SI-SDR — range: dB
  • Scale-invariant signal-to-distortion ratio, evaluates separation quality independent of amplitude scaling between prediction and target.

Input / output format

Input: A 10-second audio mixture (SNR ~0 dB) containing a target sound and background noise, paired with a text label query specifying the target sound.

Output: Separated audio waveform corresponding to the target sound class specified by the text query.

Scoring recipe

def score(pred, gold, mixture):
  sdr_out = compute_sdr(pred, gold)
  sdr_mix = compute_sdr(mixture, gold)
  sdri = sdr_out - sdr_mix
  si_sdr = compute_si_sdr(pred, gold)
  return sdri, si_sdr

Common pitfalls

• Mixtures are constructed by uniformly sampling loudness between -35 dB and -25 dB LUFS before mixing, not fixed amplitude.
• The evaluation set (VGGSound-Clean) consists of manually selected clean samples, not the full test split.

Evidence (verbatim from paper)

In a similar way to [23], we manually selected 100 clean samples that each contain a distinct target sound event from the VGGSound test set. ... Finally, we constructed an evaluation set with 1000 samples. The average SNR of the evaluation set is around 00 dB. ... We utilize signal-to-distortion ratio improvement (SDRi) and scale-invariant SDR (SI-SDR) to evaluate the performance of sound separation systems.

Citation

@misc{liu2023separate,
  title={Separate Anything You Describe},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2308.05037}
}

• arXiv: 2308.05037