Humanoid Policy Eval

humanoid-policy-eval

Humanoid Policy ~ Human Policy — Ri-Zhao Qiu et al. (arXiv:2503.13441, 2025)

What this evaluates

Evaluates a unified state-action policy's ability to perform dexterous manipulation tasks on humanoid robots. It specifically probes in-distribution (I.D.) task execution and out-of-distribution (O.O.D.) generalization across varying backgrounds, object placements, and cross-embodiment transfers.

Datasets

• Robot & Human Manipulation Demonstrations — total ?; splits: train (-1), test (-1)

Metrics

• Success rate (primary) — range: [0, 1]
  • Calculated as the number of successful autonomous skill executions divided by the total number of trials per task setting. Reported as a fraction (e.g., 42/60) or percentage.

Input / output format

Input: Egocentric visual observations processed through a frozen DinoV2 ViT-S backbone, combined with proprioceptive state inputs (joint positions or unified state-action space representations).

Output: Future action chunks representing joint positions/actions for the humanoid robot's arms and hands.

Scoring recipe

def compute_success_rate(successes, total_trials):
    return successes / total_trials

# Per task setting (I.D. or O.O.D.):
# Count successful autonomous skill executions out of N trials
# Report fraction or percentage per task

Common pitfalls

• Confusing I.D. and O.O.D. evaluation settings: I.D. closely matches training scene setups, while O.O.D. introduces novel backgrounds/object placements seen in human data but not robot data.
• Assuming co-training with human data improves in-distribution performance: the paper notes human data has minor effects on I.D. success, primarily boosting O.O.D. generalization.
• Ignoring action speed interpolation: baselines that do not slow down/interpolate human actions exhibit unstable action speeds and fail vertical grasping tasks.

Evidence (verbatim from paper)

Table 2: Success rate of autonomous skill execution. Co-training with human data (H. Data) significantly improves the Out-Of-Distribution (O.O.D.) performance with nearly 100% relative improvement on all tasks on Humanoid A.

Citation

@misc{qiuetal2025humanoidpolicy,
  title={Humanoid Policy ~ Human Policy},
  author={Ri-Zhao Qiu et al.},
  year={2025},
  note={arXiv:2503.13441}
}

• arXiv: 2503.13441