Text Queried Audio Separation Eval

text-queried-audio-separation-eval

Text-Queried Audio Source Separation via Hierarchical Modeling — Xinlei Yin et al. (arXiv:2505.21025, 2025)

What this evaluates

This benchmark evaluates a model's ability to perform text-queried audio source separation, specifically its capacity to isolate target sound events from mixed audio based on natural language instructions. It probes both acoustic fidelity (spectral and signal-level accuracy) and semantic alignment (how well the separated audio matches the textual description).

Datasets

• AudioCaps — total 500; splits: test (500)
• Clotho v2 — total 500; splits: test (500)
• FSD50K — total 500; splits: test (500)
• 3 Sets — total 1000; splits: test (1000)
• MUSIC — total 500; splits: test (500)

Metrics

• LSD (primary) — range: other
  • Quantifies the difference between spectrograms of the predicted and target samples. Lower values indicate better separation fidelity.
• PSNR — range: other
  • Measures the logarithmic ratio of the maximum possible signal power to the mean squared error between the predicted and target signals. Higher values indicate better signal reconstruction.
• KL — range: other
  • Measures the similarity between the predicted and target audio using labels calculated by an audio tagging model. Lower values indicate better alignment.
• CLAP score — range: other
  • Calculates audio-text similarity based on the MSCLAP model. Higher scores indicate better semantic alignment with the query.
• AFSim — range: other
  • Computes cosine similarity between predicted and target signals on semantic embeddings extracted from Audio Flamingo (mean-pooled penultimate layer). Higher scores indicate better semantic fidelity.

Input / output format

Input: A mixed audio waveform (typically 10 seconds, resampled to 16 kHz) containing overlapping sound events, paired with a natural language instruction describing the target audio or the undesired audio to be removed.

Output: A separated target audio waveform (same duration and sampling rate as input).

Scoring recipe

def compute_metrics(pred_audio, target_audio, query_text):
    # Spectral & Signal metrics
    lsd = np.mean(np.log10(np.abs(S_pred) / np.abs(S_target)))
    mse = np.mean((pred_audio - target_audio)**2)
    psnr = 10 * np.log10(max_power / mse)
    kl = kl_divergence(tag_model(pred_audio), tag_model(target_audio))
    # Semantic metrics
    clap = msclap_similarity(pred_audio, query_text)
    emb_pred = mean_pool(AudioFlamingo(pred_audio, prompt=query_text))
    emb_target = mean_pool(AudioFlamingo(target_audio, prompt=query_text))
    af_sim = cosine_similarity(emb_pred, emb_target)
    return {'LSD': lsd, 'PSNR': psnr, 'KL': kl, 'CLAP': clap, 'AFSim': af_sim}

Common pitfalls

• Mixing sources from different datasets introduces distribution biases; the '3 Sets' benchmark explicitly combines equal samples from three datasets to better approximate real-world recordings.
• Standard CLAP scores poorly capture temporal semantics and complex multi-source audio semantics, so the AFSim metric (using Audio Flamingo embeddings) is required for accurate semantic evaluation.
• SNR control during mixture creation varies by benchmark: [-15dB, 15dB] for general test sets, but strictly [-5dB, 5dB] for the MUSIC zero-shot dataset.

Evidence (verbatim from paper)

We evaluate our separation performance using log spectral distance (LSD), peak signal-to-noise ratio (PSNR), and Kullback-Leibler divergence (KL), following the AudioLDM_eval audio generation evaluation pipeline. LSD quantifies the difference between spectrograms of the predicted and target samples. PSNR measures the logarithmic ratio of the maximum possible signal power to the mean squared error between the predicted and target signals. KL divergence measures the similarity between the predicted and target audio with the label calculated by an audio tagging model.

Citation

@misc{yin2025textqueriedaudiosource,
  title={Text-Queried Audio Source Separation via Hierarchical Modeling},
  author={Xinlei Yin et al.},
  year={2025},
  note={arXiv:2505.21025}
}

• arXiv: 2505.21025