qhjqhj00 sound-separation-eval
Unleashing the Power of Natural Audio Featuring Multiple Sound Sources — Xize Cheng et al. (arXiv:2504.17782, 2025)
What this evaluates
Evaluates text-queried sound separation models on natural and mixed audio. It measures how accurately a model isolates target sound sources from background noise or other sources, and how well it handles silence when the target is absent.
Datasets
- AudioSet — total ?; splits: test (-1)
- AudioCaps — total ?; splits: test (-1)
- ESC-50 — total ?; splits: test (-1)
Metrics
SDRi(primary) — range: dB- Improved Signal-to-Distortion Ratio. Computes the ratio of target energy to distortion energy after applying an optimal scaling factor to the estimate to account for gain differences. Higher is better.
SISDRi— range: dB- Improved Scale-Invariant Signal-to-Distortion Ratio. Similar to SDRi but uses scale-invariant projection, making it robust to amplitude scaling differences between reference and estimate. Higher is better.
Re-SDR— range: dB- Remix-based SDR. Separated tracks are remixed back into the original mixture, then SDRi is computed between the remixed result and the original mixture to evaluate real-world separation fidelity.
Re-SISDR— range: dB- Remix-based SISDR. Same as Re-SDR but uses the scale-invariant variant for robustness to amplitude mismatches in natural audio.
Silent-SDR— range: dB- Measures separation purity when the target source is absent. Computes SDRi between the separated output and a silence signal (zeros). High values indicate minimal noise leakage.
Input / output format
Input: 10-second audio clips sampled at 32 kHz, paired with a text query (class label or caption) specifying the target source to separate.
Output: Separated audio track(s) corresponding to the queried target source.
Scoring recipe
def compute_sdr_i(reference, estimate):
# Optimal scaling for improved SDR
alpha = np.dot(reference, estimate) / (np.linalg.norm(estimate)**2 + 1e-8)
target = alpha * estimate
noise = reference - target
return 10 * np.log10(np.sum(target**2) / (np.sum(noise**2) + 1e-8))
def compute_re_sdr(separated_tracks, original_mixture):
# Remix tracks back to original shape
remixed = np.sum(separated_tracks, axis=0)
return compute_sdr_i(original_mixture, remixed)
def compute_silent_sdr(separated_output):
# Compare against silence
silence = np.zeros_like(separated_output)
return compute_sdr_i(silence, separated_output)
Common pitfalls
- Overlapping data between the AudioCaps test set and AudioSet training set must be explicitly removed before evaluation to ensure fair comparison.
- Re-SDR and Re-SISDR require remixing the separated tracks back into the original mixture before computing metrics, rather than comparing separated outputs directly to the original mixture.
- Silent-SDR evaluates purity by measuring leakage when the target source is completely absent, not standard separation quality; high values indicate successful silence preservation.
Evidence (verbatim from paper)
Following Ma et al. (2024), our primary evaluation metrics are SDRi and SISDRi, supplemented by SDR and SISDR measurements for additional experimental reference. Please note that due to some overlap between the AudioCaps test set and the AudioSet training set, we removed the overlapping data from the AudioSet training set to ensure a fair comparison.
Citation
@misc{cheng2025unleashing,
title={Unleashing the Power of Natural Audio Featuring Multiple Sound Sources},
author={Xize Cheng et al.},
year={2025},
note={arXiv:2504.17782}
}
- arXiv: 2504.17782