qhjqhj00 esc50-sep-eval
Separate Anything You Describe — Liu et al. (2023) (arXiv:2308.05037, 2023)
What this evaluates
Evaluates zero-shot language-queried audio source separation on environmental sound classes. The benchmark tests the model's ability to isolate a target sound from a mixed audio mixture using text labels.
Datasets
- ESC-50 — total 2000; splits: test (2000)
Metrics
SDRi(primary) — range: dB- Signal-to-distortion ratio improvement, calculated as the difference between the SDR of the separated output and the original mixture.
SI-SDR— range: dB- Scale-invariant signal-to-distortion ratio, evaluates separation quality independent of amplitude scaling between prediction and target.
Input / output format
Input: A 5-second audio mixture (SNR 0 dB) containing two clips from different semantic classes, paired with a text label query specifying the target class.
Output: Separated audio waveform corresponding to the target sound class specified by the text query.
Scoring recipe
def score(pred, gold, mixture):
sdr_out = compute_sdr(pred, gold)
sdr_mix = compute_sdr(mixture, gold)
sdri = sdr_out - sdr_mix
si_sdr = compute_si_sdr(pred, gold)
return sdri, si_sdr
Common pitfalls
- All clips are downsampled to 32 kHz for consistent evaluation, differing from the original 44.1 kHz.
- Mixtures are formed by randomly pairing two clips from different sound classes, not from the same class.
Evidence (verbatim from paper)
The ESC-50 dataset contains 2000 environmental audio recordings evenly arranged into 50 semantic classes. ... This leads to a total of 2000 evaluation pairs, which are used to evaluate the zero-shot performance of our model on environmental sound separation. ... We utilize signal-to-distortion ratio improvement (SDRi) and scale-invariant SDR (SI-SDR) to evaluate the performance of sound separation systems.
Citation
@misc{liu2023separate,
title={Separate Anything You Describe},
author={Liu et al. (2023)},
year={2023},
note={arXiv:2308.05037}
}
- arXiv: 2308.05037