Mmaudiosep Separation Eval

mmaudiosep-separation-eval

MMAudioSep: Taming Video-to-Audio Generative Model Towards Video/Text-Queried Sound Separation — Akira Takahashi et al. (2025) (arXiv:2510.09065, 2025)

What this evaluates

Evaluates a generative model's ability to separate target sounds from mixture audio using video and text queries, while preserving video-to-audio generation capabilities. It probes multimodal conditioning, cross-domain knowledge transfer, and the perceptual quality of generated separated audio against discriminative baselines.

Datasets

• VGGSound-Clean — total 5004; splits: test (5004); repo https://github.com/Audio-AGI/AudioSep
• MUSIC — total 1000; splits: test (1000); repo https://github.com/Audio-AGI/AudioSep
• VGGSound — total 15000; splits: test (15000)

Metrics

• FAD (primary) — range: other
  • Fréchet Audio Distance computed on PaSST features to measure feature distribution similarity between generated and reference audio. Lower values indicate better distribution matching.
• CLAPScore — range: [0, 1]
  • Semantic alignment score between output audio and text query using CLAP embeddings. Higher values indicate better text-audio alignment.
• CLAPScore_A — range: [0, 1]
  • Similarity score between output audio and ground-truth target audio using CLAP embeddings. Higher values indicate better target reconstruction.
• IB-Score — range: [0, 1]
  • Average cosine similarity of ImageBind features between input video and generated audio to measure AV semantic alignment.
• DeSync — range: other
  • Temporal misalignment predicted by Synchformer features between input video and generated audio. Lower values indicate better synchronization.
• IS — range: other
  • Inception Score calculated with a PANNs classifier to evaluate the quality and diversity of generated audio.
• FD — range: other
  • Fréchet Distance computed on PaSST features for distribution matching between generated and reference audio.
• KL — range: other
  • Kullback-Leibler divergence computed on PaSST features for distribution matching between generated and reference audio.

Input / output format

Input: 10-second mixture audio, 10-second video query (padded with black frames if shorter), and text query (class or instrument label).

Output: 10-second separated target audio (for separation task) or 8-second generated audio (for V2A task).

Scoring recipe

def evaluate_separation(pred_audio, gold_audio, text_query, video_query):
    # Compute generative metrics as specified in the paper
    fad = frechet_audio_distance(pred_audio, gold_audio, model='PaSST')
    clap_score = clap_similarity(pred_audio, text_query)
    clap_a = clap_similarity(pred_audio, gold_audio)
    ib_score = cosine_similarity(imagebind(pred_audio), imagebind(video_query))
    desync = synchformer_misalignment(pred_audio, video_query)
    is_score = inception_score(pred_audio, classifier='PANNs')
    fd = frechet_distance(pred_audio, gold_audio, model='PaSST')
    kl = kl_divergence(pred_audio, gold_audio, model='PaSST')
    return {'FAD': fad, 'CLAPScore': clap_score, 'CLAPScore_A': clap_a,
            'IB-Score': ib_score, 'DeSync': desync, 'IS': is_score,
            'FD': fd, 'KL': kl}

Common pitfalls

• Using discriminative metrics like SDR instead of generative metrics (FAD, CLAP), which the authors explicitly note are suboptimal for this generative approach.
• Including PANNs in FD/KL computation, as the paper explicitly excludes it due to reported robustness issues in some scenarios.
• Failing to pad videos to exactly 10 seconds, which breaks temporal consistency between audio and video inputs required for the evaluation protocol.

Evidence (verbatim from paper)

As MMAudioSep is a generative-based approach, conventional sample-level objective metrics for sound separation tasks, such as the source-to-distortion ratio (SDR), are suboptimal for assessing the proposed system. Instead, we follow the evaluation protocol of FlowSep [[5]], employing metrics suited for generative tasks. These include Fréchet Audio Distance (FAD) [[28]] for feature distribution similarity and CLAP-based scores [[29]]. Specifically, CLAPScore evaluates the semantic alignment between the output audio and the text query, while CLAPScore ${}_{\textbf{A}}$ measures the similarity to the ground-truth target audio.

Citation

@misc{takahashi2025mmaudiosep,
  title={MMAudioSep: Taming Video-to-Audio Generative Model Towards Video/Text-Queried Sound Separation},
  author={Akira Takahashi et al. (2025)},
  year={2025},
  note={arXiv:2510.09065}
}

• arXiv: 2510.09065