Humanoid Everyday Eval

humanoid-everyday-eval

Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation — Zhao et al. (2025) (arXiv:2510.08807, 2025)

What this evaluates

Evaluates imitation learning and vision-language-action policies on open-world humanoid manipulation. It probes robustness to high-dimensional action spaces, multimodal sensor fusion, and fine-grained visuospatial perception across locomotion, tool use, and precise manipulation tasks.

Datasets

• Humanoid Everyday — total 10300; splits: train (-1), test (-1)

Metrics

• success rate (primary) — range: percent
  • Calculated as the number of successful task completions divided by the total number of trials (10 per task), multiplied by 100.

Input / output format

Input: Multimodal sensor observations (RGB, depth, LiDAR, tactile, IMU) and natural language task instructions, synchronized at 30Hz.

Output: 28-degree-of-freedom (DoF) robot action commands.

Scoring recipe

def compute_success_rate(successful_trials, total_trials=10):
    return (successful_trials / total_trials) * 100
# Evaluated over 10 trials per task. Average success rate is computed across all 7 tasks.

Common pitfalls

• High-dimensional action space (28 DoFs) causes most end-to-end imitation policies to struggle significantly.
• Downsampling VLA model actions from 30Hz to 2Hz alleviates decoding failures but reduces motion smoothness.
• 3D point cloud inputs become unreliable in locomotion tasks due to large frame-to-frame camera changes.
• Fine-grained visuospatial perception is required for high-precision tasks, which current models largely lack.

Evidence (verbatim from paper)

We present the results of these policies in Table II. where each policy is evaluated over 10 trials on seven tasks. TABLE II: Success rates of imitation learning methods on the Humanoid Everyday Dataset.

Citation

@misc{zhao2025humanoid,
  title={Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2510.08807}
}

• arXiv: 2510.08807