Beat It Eval

beat-it-eval

Beat-It: Beat-Synchronized Multi-Condition 3D Dance Generation — Huang et al. (2024) (arXiv:2407.07554, 2024)

What this evaluates

Evaluates a model's ability to generate 3D dance motions that are temporally synchronized with musical beats and controllable via sparse keyframes, while maintaining kinematic plausibility and motion diversity.

Datasets

• AIST++ — total 1408; splits: test (-1)

Metrics

• PFC — range: other
  • Physical Foot Contact score measuring kinematic plausibility by evaluating foot contact consistency in generated motions.
• BAS (primary) — range: other
  • Beat Alignment Score quantifying the synchronization quality between the generated dance and the musical beat.
• Div_k — range: other
  • Average feature distance of kinetic features extracted by fairmotion between generated and ground truth motions.
• Div_g — range: other
  • Average feature distance of geometric features extracted by fairmotion between generated and ground truth motions.
• KPD — range: other
  • Average mean squared error of local joint positions in Cartesian space at the specified keyframes.
• BAP — range: percent
  • Percentage of beats in the generated dance that are correctly assigned to the designated beat frames.

Input / output format

Input: 5-second audio clips (30 fps motion) with 2.5s overlap, sparse keyframe conditions (10% of frames), and designated beat conditions extracted via Librosa.

Output: 3D dance motion sequences (5-second clips at 30 fps).

Scoring recipe

def compute_metrics(pred_motions, gt_motions, keyframes, beats):
    pfc = physical_foot_contact(pred_motions)  # lower is better
    bas = beat_alignment_score(pred_motions, beats)  # higher is better
    div_k = feature_distance(pred_motions, gt_motions, type='kinetic')  # closer to GT is better
    div_g = feature_distance(pred_motions, gt_motions, type='geometric')  # closer to GT is better
    kpd = mean_squared_error(pred_motions[keyframes], gt_motions[keyframes])  # lower is better
    bap = percentage_correct_beat_assignment(pred_motions, beats)  # higher is better
    return {'PFC': pfc, 'BAS': bas, 'Div_k': div_k, 'Div_g': div_g, 'KPD': kpd, 'BAP': bap}

Common pitfalls

• FID is explicitly deemed unreliable for this dataset due to limited test samples and should not be used.
• Data leakage must be prevented by randomly sampling keyframes and beat conditions from unpaired test samples rather than using ground truth conditions.
• Metrics like BAS and BAP measure different aspects of beat alignment; a high BAS does not guarantee adherence to specific beat choreography constraints.

Evidence (verbatim from paper)

For quantitative evaluation, we measure the generated dance from three aspects: generation quality, diversity, and controllability. Due to the limited samples in the testing set of AIST++ [23], the prior work EDGE [42] has demonstrated that the Fréchet Inception Distance (FID) is not reliable for measuring the generation quality. Therefore, we do not adopt FID as the evaluation metric. Instead, we assess generation quality from two different perspectives: music-dance correlation and kinematic plausibility. In terms of music-dance correlation, we utilize the Beat Alignment Score (BAS) metric, following the approach presented in [35]. BAS quantifies the synchronization quality between the generated dance and the musical beat. For the evaluation of kinematic plausibility, we adopt the Physical Foot Contact score (PFC) metric proposed in [42]. PFC provides a metric for measuring the physical plausibility of the generated dance from a kinematic perspective. As for the diversity, we measure the average feature distance of kinetic (Div_k) and geometric (Div_g) features extracted by fairmotion [12] following the previous works [23, 35, 42]. For assessing controllability, we employ the Key

Citation

@misc{huang2024beatit,
  title={Beat-It: Beat-Synchronized Multi-Condition 3D Dance Generation},
  author={Huang et al. (2024)},
  year={2024},
  note={arXiv:2407.07554}
}

• arXiv: 2407.07554