Loris Eval

loris-eval

Long-Term Rhythmic Video Soundtracker — Jiashuo Yu et al. (2023) (arXiv:2305.01319, 2023)

What this evaluates

Evaluates a model's ability to generate long-term (25s–50s) high-fidelity musical waveforms that are rhythmically synchronized with visual cues from diverse video scenarios like dancing and sports. It measures both the temporal alignment of generated beats with ground-truth audio and the overall subjective musical quality.

Datasets

• LORIS — total 17253; splits: train (-1), val (-1), test (-1); repo https://github.com/OpenGVLab/LORIS

Metrics

• BCS — range: percent
  • Aligned beats divided by generated beats ($B_a / B_g$). Acts as a recall metric for rhythm coverage.
• BHS — range: percent
  • Aligned beats divided by ground-truth beats ($B_a / B_t$). Acts as a precision metric for rhythm alignment.
• F1 (primary) — range: percent
  • Harmonic mean of BCS and BHS: $2 imes (BCS imes BHS) / (BCS + BHS)$. Serves as the integrated assessment metric.
• CSD — range: other
  • Standard deviation of BCS scores across a batch, measuring generative stability.
• HSD — range: other
  • Standard deviation of BHS scores across a batch, measuring generative stability.
• MOS — range: other
  • Mean Opinion Score from subjective human evaluation of general audio quality.

Input / output format

Input: RGB video frames (or extracted I3D features) and optionally 2D skeleton poses, along with categorical genre labels. Visual rhythm features are extracted via a Bi-LSTM and Hawkes process.

Output: A 22kHz audio waveform of 25 or 50 seconds duration.

Scoring recipe

# Detect beats in generated (B_g) and ground-truth (B_t) audio
B_g = detect_beats(generated_audio)
B_t = detect_beats(ground_truth_audio)
# Find aligned beats (B_a) based on temporal proximity
B_a = count_aligned_beats(B_g, B_t)

BCS = B_a / B_g
BHS = B_a / B_t
F1 = 2 * (BCS * BHS) / (BCS + BHS)

# For a batch of N samples:
CSD = std([BCS_i for i in range(N)])
HSD = std([BHS_i for i in range(N)])

Common pitfalls

• Original BCS/BHS formulas ($B_g/B_t$ and $B_a/B_t$) are unsuitable for long-term audio as they yield sparse vectors and allow BHS > 1, masking poor performance.
• Audio onset detection parameters must be explicitly tuned for long sequences; default settings produce artificially low scores.
• MOS requires human raters and cannot be computed automatically from audio files.

Evidence (verbatim from paper)

To investigate rhythm correspondence, we use the improved versions of beats coverage scores (BCS) and beats hit scores (BHS) for evaluation. To be specific, BCS and BHS are first proposed for music-guided dance generation (Davis & Agrawala, 2018; Lee et al., 2019) which measures the alignment of musical rhythms and dancing patterns. Similarly, prior dance-to-music methods (Zhu et al., 2022a;b) employ these metrics to count the aligned rhythm points of synthesized music and ground-truth music by computing the rhythm point number of generated music $B_{g}$ , the rhythm point number of ground-truth music $B_{t}$ , and the number of aligned rhythm points $B_{a}$ . Then, BCS is calculated as the fraction of generated musical beats by the ground truth musical beats $(B_{g} / B_{t})$ , and BHS measures the ratio of aligned beats to the ground truth beats $(B_{a} / B_{t})$ . However, we found that these metrics are only suitable for short-length (2~6s) music, and two main problems emerge when evaluating long-term soundtracks: 1). the second-wise rhythm detection algorithm results in an extremely sparse vector for any long music sequence, thus the constantly low BCS and BHS values are

Citation

@misc{yu2023longtermrhythmicvideosoundtracker,
  title={Long-Term Rhythmic Video Soundtracker},
  author={Jiashuo Yu et al. (2023)},
  year={2023},
  note={arXiv:2305.01319}
}

• arXiv: 2305.01319