Vidmuse Video To Music Eval

vidmuse-video-to-music-eval

VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling — Tian et al. (2024) (arXiv:2406.04321, 2024)

What this evaluates

Evaluates a model's ability to generate high-fidelity, semantically aligned music conditioned on video input. It probes audio quality, diversity, and cross-modal alignment using statistical distance metrics, beat alignment scores, and subjective human preference tests.

Datasets

• V2M — total ?; splits: test (-1)
• AIST++ — total ?; splits: test (-1)
• LORIS — total ?; splits: test (-1)
• TikTok — total ?; splits: test (-1)

Metrics

• FAD (primary) — range: [0, inf)
  • Frechet Audio Distance; measures the distance between feature distributions of generated and real audio in a pretrained feature space. Lower is better.
• ImageBind Score — range: [0, 1]
  • Cross-modal alignment score computed using the ImageBind model to measure semantic consistency between video and generated music. Higher is better.
• BCS — range: [0, 100]
  • Beats Coverage Score; measures the proportion of ground truth beats covered by generated music beats. Higher is better.
• BHS — range: [0, 100]
  • Beats Hit Score; measures the proportion of generated beats that match ground truth beats. Higher is better.
• Density — range: [0, inf)
  • Diversity metric indicating the average pairwise distance between generated samples. Higher indicates greater diversity.
• Coverage — range: [0, 1]
  • Diversity metric indicating the fraction of ground truth samples covered by generated samples. Higher indicates better coverage.

Input / output format

Input: 30-second video segments; frames sampled at 2 fps for short-term modeling and 32 uniformly sampled frames for long-term modeling.

Output: Autoregressive music tokens sampled via top-k (k=250, temp=1.0), decoded into 32 kHz monophonic audio waveforms.

Scoring recipe

def evaluate(predictions, gold):
    fad = frechet_audio_distance(predictions, gold)
    img_score = imagebind_alignment(predictions, gold)
    density, coverage = diversity_metrics(predictions)
    bcs, bhs = beat_coverage_hit(predictions, gold)
    return {'FAD': fad, 'ImageBind': img_score, 'Density': density, 'Coverage': coverage, 'BCS': bcs, 'BHS': bhs}

Common pitfalls

• ImageBind Score is not trained on music data, so it may poorly capture semantic alignment for audio.
• BCS/BHS metrics are borrowed from non-public baselines and may lack standardized open-source implementations.
• Sliding window inference (30s window, 0.5s overlap) can introduce boundary artifacts that skew diversity metrics if not evaluated on full untrimmed generations.

Evidence (verbatim from paper)

To quantitatively evaluate the effectiveness of our model, we employ a series of metrics to assess different models in terms of quality, fidelity, and diversity of the generated music. These metrics include the Frechet Audio Distance (FAD), Frechet Distance (FD), Kullback-Leibler Divergence (KL), as well as Density and Coverage*[[55]]. Additionally, we utilize the ImageBind Score[[22]]* to examine the alignment between the video and the generated music.

Citation

@misc{tian2024vidmuse,
  title={VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling},
  author={Tian et al. (2024)},
  year={2024},
  note={arXiv:2406.04321}
}

• arXiv: 2406.04321