Video To Music Eval

video-to-music-eval

Video-Robin: Autoregressive Diffusion Planning for Intent-Grounded Video-to-Music Generation — Lokegaonkar et al. (2026) (arXiv:2604.17656, 2026)

What this evaluates

This evaluation protocol assesses a model's ability to generate high-fidelity, diverse instrumental music that is semantically and temporally aligned with a given 10-second video and optional fine-grained text prompt. It probes audio quality, distributional fidelity, generative diversity, and cross-modal alignment using both automated perceptual metrics and human/LLM preference judgments.

Datasets

• ReelBench — total ?; splits: test (-1)
• LORIS — total ?; splits: test (-1)
• V2MBench — total ?; splits: test (-1)

Metrics

• FAD (primary) — range: other
  • Frechet Audio Distance measures the L2 distance between the mean and covariance of real and generated audio features in a learned embedding space. Lower values indicate higher perceptual quality and distributional alignment.
• FD — range: other
  • Frechet Distance between feature distributions of real and generated audio, assessing overall distributional alignment in the embedding space.
• KL — range: other
  • Kullback-Leibler Divergence between the probability distributions of real and generated audio features. Lower values indicate closer distributional match.
• IS — range: other
  • Inception Score combines the mean KL divergence of class predictions conditioned on generated samples and the marginal class distribution, measuring perceptual quality and diversity.
• IB — range: other
  • ImageBind Score measures cross-modal alignment between video and audio embeddings in the ImageBind space via cosine similarity.
• Density & Coverage — range: [0, 1]
  • Density measures the fraction of real samples that have a generated neighbor within a fixed distance threshold; Coverage measures the fraction of generated samples that have a real neighbor. Higher values indicate better distributional coverage.

Input / output format

Input: 10-second video sequence (truncated if longer) paired with a fine-grained text prompt describing musical attributes (tempo, key, chord progression). If no prompt is available, a static instruction 'Generate aligned music for the video.' is used.

Output: 10-second instrumental music audio track at 48 kHz stereo.

Scoring recipe

def compute_metrics(gold_audio, pred_audio, video_input):
    real_feats = extract_audio_features(gold_audio)
    gen_feats = extract_audio_features(pred_audio)
    video_feats = extract_video_features(video_input)
    
    fad = frechet_audio_distance(real_feats, gen_feats)
    fd = frechet_distance(real_feats, gen_feats)
    kl = kl_divergence(real_feats, gen_feats)
    is_score = inception_score(gen_feats)
    density, coverage = density_coverage(real_feats, gen_feats, k=10)
    ib_score = cosine_similarity(video_feats, gen_feats)
    
    return {'FAD': fad, 'FD': fd, 'KL': kl, 'IS': is_score, 'IB': ib_score, 'Density': density, 'Coverage': coverage}

Common pitfalls

• Ground-truth audio must have vocals stripped and non-musical artifacts removed before evaluation; otherwise, FAD/FD scores will be artificially inflated.
• Videos must be strictly truncated to 10 seconds to match the model's fixed-length generation setting; longer inputs cause misalignment in temporal metrics.
• ImageBind Score alone is insufficient for alignment evaluation as the model lacks music-specific training; the paper supplements it with a Gemini Omni-Judge across 7 axes.

Evidence (verbatim from paper)

We quantitatively evaluate the effectiveness of our model using previously adopted metrics for music generation. These metrics evaluate the quality, fidelity and diversity of the generated music. We use Frechet Audio Distance (FAD), Frechet Distance (FD), KL Divergence (KL), Inception Score (IS), ImageBind Score (IB), Density, and Coverage. The quality of the generated music is measured by FAD, FD, and KL, while its alignment with the video is measured by the ImageBind Score.

Citation

@misc{lokegaonkar2026videorobin,
  title={Video-Robin: Autoregressive Diffusion Planning for Intent-Grounded Video-to-Music Generation},
  author={Lokegaonkar et al. (2026)},
  year={2026},
  note={arXiv:2604.17656}
}

• arXiv: 2604.17656