Humanoid Pose Control Eval

humanoid-pose-control-eval

Learning from Massive Human Videos for Universal Humanoid Pose Control — Jiageng Mao et al. (arXiv:2412.14172, 2024)

What this evaluates

Evaluates a language-conditioned transformer model's ability to generate physically plausible and text-aligned 3D humanoid poses from text commands. It probes motion quality, diversity, and multimodal alignment on a retargeted human motion benchmark, as well as real-world deployment success rates.

Datasets

• HumanoidML3D — total ?; splits: test (-1)
• Humanoid-X — total 20000000; splits: train (20000000)

Metrics

• FID (primary) — range: other
  • Frechet Inception Distance measuring the dissimilarity between feature distributions of generated and ground-truth humanoid poses. Lower is better.
• MM Dist — range: other
  • Multi-modal Distance measuring the Euclidean distance between generated motions and corresponding text embeddings. Lower is better.
• Diversity — range: other
  • Average Euclidean distance between 300 randomly sampled pairs of generated humanoid poses. Higher is better.
• R Precision — range: [0, 1]
  • Accuracy of matching generated poses to their corresponding text descriptions within the top-3 ranked candidates. Higher is better.

Input / output format

Input: Natural language text commands describing desired humanoid actions.

Output: 3D humanoid keypoint sequences or direct robotic joint action sequences.

Scoring recipe

def compute_metrics(predictions, ground_truth, texts):
    fid = frechet_inception_distance(predictions.features, ground_truth.features)
    diversity = mean_pairwise_euclidean_distance(predictions, n_samples=300)
    mm_dist = mean_euclidean_distance(predictions.features, texts.embeddings)
    r_prec = top_k_accuracy(predictions.features, texts, k=3)
    return {'FID': fid, 'Diversity': diversity, 'MM Dist': mm_dist, 'R Precision': r_prec}

Common pitfalls

• FID and MM Dist are highly sensitive to the choice of feature extractor and text encoder, making cross-paper comparisons difficult without identical backbones.
• Diversity is calculated over only 300 random pairs, which can introduce high variance if the generation distribution is multi-modal or sparse.
• Real-world success rates depend heavily on the control mode (open-loop vs. closed-loop) and the separation of upper/lower body control policies, which are not captured by offline metrics.

Evidence (verbatim from paper)

We adopt the metrics in [18] to evaluate the humanoid motions from different aspects: (1) Quality: The Frechet Inception Distance (FID) evaluates the dissimilarity between feature distributions of generated and ground truth humanoid poses. (2) Diversity: The Diversity metric evaluates the variability within the generated humanoid pose distribution, calculated as the average Euclidean distance between 300 randomly sampled pairs of humanoid poses. (3) Reliability: The Multi-modal Distance (MM Dist) measures the Euclidean distance between motions and corresponding texts, and the $R$ Precision assesses the accuracy of text and humanoid pose matches in the Top 3 rankings.

Citation

@misc{mao2024humanoidx,
  title={Learning from Massive Human Videos for Universal Humanoid Pose Control},
  author={Jiageng Mao et al.},
  year={2024},
  note={arXiv:2412.14172}
}

• arXiv: 2412.14172