Sam Audio Eval

sam-audio-eval

SAM Audio: Segment Anything in Audio — Shi et al. (2025) (arXiv:2512.18099, 2025)

What this evaluates

Evaluates audio source separation capabilities conditioned on text, visual masks, or temporal spans. It probes open-vocabulary extraction, speaker/music/instrument isolation, and cross-modal grounding in both studio and in-the-wild settings.

Datasets

• SAM Audio Evaluation Set — total ?; splits: test (-1)
• MUSDB18 — total ?; splits: test (-1)

Metrics

• separation fidelity (primary) — range: other
  • Not explicitly defined in the provided text; refers to the model's ability to isolate the target source without perceptual enhancement or reverb suppression.
• subjective evaluation — range: other
  • Human ratings on ~10s filtered clips to assess separation quality and prompt alignment.
• CLAP similarity — range: [0, 1]
  • Cosine similarity between audio and text embeddings using the CLAP model, used for speaker separation baseline comparison and inference re-ranking.
• span IoU — range: [0, 1]
  • Intersection over Union between predicted and ground-truth temporal spans for span-prompted tasks.

Input / output format

Input: Audio mixture clip (~10s or up to 30s) paired with conditioning: a text prompt, a pair of video masks with the raw video, or temporal spans indicating target source activity.

Output: Separated audio track/stem corresponding to the target source specified by the prompt.

Scoring recipe

def evaluate(audio_mixture, prompt, ground_truth):
    pred_stem = model.predict(audio_mixture, prompt)
    fidelity = compute_separation_fidelity(pred_stem, ground_truth)
    subjective = human_rate(pred_stem)
    
    # Inference re-ranking scores (not final eval metrics)
    if prompt.type == 'text':
        rank = 1.0 * clap_score(pred_stem, prompt.text) + 5.0 * judge_score(pred_stem)
    elif prompt.type == 'span':
        rank = compute_iou(pred_stem.span, ground_truth.span)
    elif prompt.type == 'visual':
        rank = imagebind_similarity(pred_stem, prompt.video_mask)
    return fidelity, subjective, rank

Common pitfalls

• Baselines are proprietary APIs; the paper explicitly disables their post-processing/enhancement modules to isolate separation fidelity, which may differ from standard API usage.
• Visual-prompted baselines (e.g., AV-Mossformer2) fail preprocessing on ~20% of evaluation videos, requiring sample exclusion that isn't always transparent.
• Evaluation clips are truncated to ~10s for subjective testing, which may not capture long-context separation performance.

Evidence (verbatim from paper)

To ensure fair comparison, we disable post-processing or enhancement modules in the baselines, as our evaluation focuses on separation fidelity—the model's ability to isolate speech content—rather than perceptual enhancement or reverb suppression.

Citation

@misc{shi2025samaudio,
  title={SAM Audio: Segment Anything in Audio},
  author={Shi et al. (2025)},
  year={2025},
  note={arXiv:2512.18099}
}

• arXiv: 2512.18099