Phuma Eval

phuma-eval

PHUMA: Physically-Grounded Humanoid Locomotion Dataset — Kyungmin Lee et al. (arXiv:2510.26236, 2025)

What this evaluates

Evaluates a humanoid robot's ability to imitate human motion and follow pelvis trajectories using physically-grounded retargeting. It probes full-body tracking accuracy and partial-state path-following control across diverse locomotion categories on Unitree G1 and H1-2 robots.

Datasets

• PHUMA — total ?; splits: train (-1), test (7500)
• Unseen Video — total 504; splits: test (504)

Metrics

• success_rate (primary) — range: percent
  • Ratio of motion sequences where the maximum deviation between the tracked trajectory and the target trajectory remains within a 0.15m threshold throughout the sequence.

Input / output format

Input: Full-state: current proprioceptive state (joint positions, orientations, velocities) and full goal states (target motion trajectories). Partial-state: pelvis position and rotation.

Output: Joint angle commands executed via PD controllers.

Scoring recipe

def compute_success_rate(predictions, gold, threshold=0.15):
    successful = 0
    for motion_idx in range(len(predictions)):
        max_deviation = max(distance(predictions[motion_idx][t], gold[motion_idx][t]) for t in range(len(predictions[motion_idx])))
        if max_deviation <= threshold:
            successful += 1
    return (successful / len(predictions)) * 100

Common pitfalls

• Using a 0.5m threshold incorrectly classifies scenarios as successful when humanoids remain stationary during jumps or stay upright during squatting motions.
• Failing to categorize motions into stationary, angular, vertical, and horizontal masks performance gaps in dynamic locomotion types.

Evidence (verbatim from paper)

For evaluating the full body motion tracking (RQ1, RQ2), we adopt the success rate metric from prior motion imitation studies (He et al., 2024b; 2025a; Xie et al., 2025), which measures the ratio of motions successfully imitated within a specified deviation threshold. Unlike prior work that uses a 0.5m threshold, we employ a stricter 0.15m threshold, as the standard threshold incorrectly classifies scenarios as successful when humanoids remain stationary during jumps or stay upright during squatting motions.

Citation

@misc{lee2025phuma,
  title={PHUMA: Physically-Grounded Humanoid Locomotion Dataset},
  author={Kyungmin Lee et al.},
  year={2025},
  note={arXiv:2510.26236}
}

• arXiv: 2510.26236