Clapsep Eval

clapsep-eval

CLAPSep: Leveraging Contrastive Pre-trained Model for Multi-Modal Query-Conditioned Target Sound Extraction — Hao Ma et al. (2024) (arXiv:2402.17455, 2024)

What this evaluates

This benchmark evaluates query-conditioned target sound extraction (TSE), testing a model's ability to isolate a target audio source from a mixture using language captions or reference audio queries. It probes multi-modal query processing and positive/negative query valence across diverse acoustic environments and musical instruments.

Datasets

• AudioCaps — total ?; splits: test (4785)
• AudioSet — total ?; splits: eval (18869)
• ESC-50 — total ?; splits: test (6500)
• FSDKaggle2018 — total ?; splits: test (8000)
• MUSIC21 — total ?; splits: test (19805)

Metrics

• SDRi (primary) — range: other
  • SDRi = SDR(estimated_source, ground_truth) - SDR(mixture, ground_truth). Measures the improvement in signal-to-distortion ratio after separation.
• SISDRi (primary) — range: other
  • SISDRi = SISDR(estimated_source, ground_truth) - SISDR(mixture, ground_truth). Measures the improvement in scale-invariant signal-to-distortion ratio after separation.

Input / output format

Input: A mixed audio waveform (target source + interference noise at 0 dB SNR) paired with a query, which can be either a text caption or a reference audio sample.

Output: A single extracted target sound source waveform corresponding to the query.

Scoring recipe

def score(predictions, gold, mixture):
    sdri = sdr(predictions, gold) - sdr(mixture, gold)
    sisdri = sisdr(predictions, gold) - sisdr(mixture, gold)
    return {'SDRi': sdri, 'SISDRi': sisdri}

Common pitfalls

• Using evaluation mixtures with SNR levels other than the strictly specified 0 dB.
• Allowing query audio samples to be used in generating the evaluation mixtures, causing information leakage.
• Failing to resample all audio to 32kHz for consistent evaluation across datasets like ESC-50 and FSDKaggle2018.
• Confusing positive vs. negative query valence, which significantly impacts performance reporting.

Evidence (verbatim from paper)

Following previous works [[19], [20]], we use signal-to-distortion ratio improvement (SDRi) and scale-invariant signal-to-distortion ratio improvement (SISDRi) as the evaluation metrics. They indicate to what extent SDR and SISDR ... are improved by sound separation. They are defined as follows, SDRi(x̂, x̃, x) = SDR(x̂, x) - SDR(x̃, x), SISDRi(x̂, x̃, x) = SISDR(x̂, x) - SISDR(x̃, x), where x̂, x̃ and x denote the extracted sound source, sound mixture, and ground truth source, respectively.

Citation

@misc{ma2024clapsep,
  title={CLAPSep: Leveraging Contrastive Pre-trained Model for Multi-Modal Query-Conditioned Target Sound Extraction},
  author={Hao Ma et al. (2024)},
  year={2024},
  note={arXiv:2402.17455}
}

• arXiv: 2402.17455