Motionbank Eval

motionbank-eval

MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotations — Liang Xu et al. (2024) (arXiv:2410.13790, 2024)

What this evaluates

Evaluates the effectiveness of the MotionBank dataset for downstream text-to-motion generation tasks. It measures how well rule-based, disentangled motion annotations improve single human motion synthesis and human-object interaction generation compared to baseline models.

Datasets

• MotionBank — total 1240000; splits: test (-1); repo https://github.com/liangxuy/MotionBank
• HumanML3D — total 14616; splits: test (-1)
• BEHAVE — total ?; splits: test (-1)

Metrics

• R Precision (primary) — range: [0, 1]
  • Measures retrieval accuracy for top-1, top-2, and top-3 matches. For each text prompt, the model retrieves the top-k motions; R Precision is the fraction of ground-truth motions correctly ranked within the top-k.
• FID — range: other
  • Frechet Inception Distance. Computes the Fréchet distance between the multivariate Gaussian distributions of real and generated motion embeddings to gauge distributional similarity.
• Diversity — range: other
  • Assesses the latent variance of all generated motion sequences, indicating the model's ability to produce varied outputs.
• MModality — range: other
  • Appraises the diversity of motions generated from the same text prompt, measuring the model's multimodal generation capability.
• MMDist — range: other
  • Computes the latent discrepancy between generated motions and their corresponding text prompts, measuring text-motion alignment in the embedding space.

Input / output format

Input: Text prompts (rule-based descriptions or natural language captions) and optional context for human-object interactions.

Output: Motion sequences represented as 263-dimensional pose vectors.

Scoring recipe

def evaluate(predictions, golds, texts, encoder):
    # R Precision: retrieval accuracy over top-1/2/3
    r_prec = compute_retrieval_accuracy(predictions, golds, k=3)
    # FID: distribution distance between real and generated motions
    fid = frechet_inception_distance(golds, predictions)
    # Diversity: latent variance of all generated motions
    diversity = torch.var(torch.stack([encoder(m) for m in predictions]))
    # MModality: variance of motions generated from identical texts
    mmodality = compute_multimodal_diversity(predictions, texts)
    # MMDist: latent distance between generated motions and their texts
    mmdist = compute_text_motion_latent_distance(predictions, texts, encoder)
    return {'R Precision': r_prec, 'FID': fid, 'Diversity': diversity, 'MModality': mmodality, 'MMDist': mmdist}

Common pitfalls

• Metrics are computed on models finetuned on HumanML3D/BEHAVE after pre-training on MotionBank, so they measure dataset utility for fine-tuning rather than zero-shot generation quality.
• Diversity, MModality, and MMDist depend on the latent space of the T2M encoder, making them architecture-sensitive and not directly comparable across different motion models.
• R Precision uses 31 randomly mismatched descriptions per query, introducing variance across different random seeds.

Evidence (verbatim from paper)

Similar to previous works, we employ the R Precision to quantify the accuracy of top-1, top-2 and top-3 retrieval from 31 randomly mismatched descriptions against the ground-truth description, the Frechet Inception Distance (FID) to gauge the distribution distance between real and generated samples, the diversity as a metric for assessing latent variance. Besides, multimodality (MModality) and MultiModal distance (MMDist) are harnessed to appraise the diversity of motions generated from the same text, and to compute the latent discrepancy between generated motions and texts, respectively.

Citation

@misc{xu2024motionbank,
  title={MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotations},
  author={Liang Xu et al. (2024)},
  year={2024},
  note={arXiv:2410.13790}
}

• arXiv: 2410.13790