Holistic Motion2d Eval

holistic-motion2d-eval

Holistic-Motion2D: Scalable Whole-body Human Motion Generation in 2D Space — Wang et al. (2024) (arXiv:2406.11253, 2024)

What this evaluates

Evaluates the quality, text-motion alignment, and diversity of generated 2D whole-body human motion sequences conditioned on text prompts. It probes the model's ability to capture fine-grained spatial-temporal dynamics and handle occlusions or noisy 2D pose data.

Datasets

• Holistic-Motion2D — total ?; splits: D1 (-1), D2 (-1), D3 (-1)

Metrics

• FID (primary) — range: other
  • Fréchet Inception Distance measuring the distance between feature distributions of real and generated motion sequences. Lower values indicate higher fidelity and realism.
• R-Precision — range: [0, 1]
  • Retrieval precision: given a text prompt, the model retrieves top-k motion sequences from a candidate pool. R-Precision indicates whether the ground-truth motion is ranked in the top-k results (typically k=3). Higher is better.
• Top1 — range: [0, 1]
  • Retrieval precision at k=1. Indicates whether the ground-truth motion is ranked first among candidates. Higher is better.
• Top3 — range: [0, 1]
  • Retrieval precision at k=3. Indicates whether the ground-truth motion is ranked in the top-3 among candidates. Higher is better.
• Diversity — range: other
  • Average pairwise distance between generated motion sequences, measuring the variety of outputs. Higher is better.

Input / output format

Input: Text prompt describing a human action or motion.

Output: 2D whole-body human motion sequence (keypoint coordinates over time).

Scoring recipe

def compute_metrics(real_motions, generated_motions, prompts, candidates, ground_truth_indices):
    # FID
    real_feats = feature_extractor(real_motions)
    gen_feats = feature_extractor(generated_motions)
    fid = frechet_distance(real_feats.mean(), real_feats.cov(), gen_feats.mean(), gen_feats.cov())
    
    # Retrieval (Top1/Top3/R-Precision)
    text_feats = text_encoder(prompts)
    motion_feats = motion_encoder(candidates)
    scores = cosine_similarity(text_feats, motion_feats)
    top_k = argsort(scores, k=3)
    r_prec = mean([1.0 if gt in top_k[i] else 0.0 for i, gt in enumerate(ground_truth_indices)])
    
    # Diversity
    diversity = mean_pairwise_distance(generated_motions)
    return {'FID': fid, 'Top1': r_prec_at_1, 'Top3': r_prec_at_3, 'Diversity': diversity}

Common pitfalls

• FID requires a fixed feature extractor and consistent motion representation; mismatched preprocessing or dimensionality skews results.
• R-Precision/Top1/Top3 scores are highly sensitive to the retrieval pool size, quality, and whether ground-truth motions are properly excluded from candidates during retrieval.
• Diversity is not explicitly defined in the paper; implementations vary (e.g., mean pairwise distance vs. variance), making cross-paper comparisons difficult.
• Low-confidence keypoints and occlusions in 2D data can artificially degrade FID or retrieval accuracy if evaluation does not account for pose confidence or filtering.

Evidence (verbatim from paper)

We achieve the best performance in FID and R-Precision, outperforming other three baseline methods on generation quality and text-motion alignment. Table 3 reports Top1, FID, and Diversity across different data scales.

Citation

@misc{wang2024holisticmotion2d,
  title={Holistic-Motion2D: Scalable Whole-body Human Motion Generation in 2D Space},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2406.11253}
}

• arXiv: 2406.11253