qhjqhj00 being-h05-robot-eval
Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalization — Hao Luo et al. (2026) (arXiv:2601.12993, 2026)
What this evaluates
Evaluates cross-embodiment generalization and manipulation capabilities of Vision-Language-Action models across heterogeneous real robots and simulation benchmarks. Probes spatial reasoning, long-horizon planning, bimanual coordination, and zero-shot transfer to unseen task-embodiment pairs.
Datasets
- Real-robot task suite — total ?; splits: test (-1)
- LIBERO — total ?; splits: test (-1)
- RoboCasa — total ?; splits: test (-1)
Metrics
success rate (%)(primary) — range: percent- Binary success/failure per trial based on pre-defined objective criteria. Reported as the mean success rate over K trials (K=20 for real robots, 500 per suite for LIBERO, 50 per task for RoboCasa).
Input / output format
Input: Multi-view RGB images (224x224 resolution), natural language task instructions, and robot proprioceptive/sensory observations.
Output: Unified action space tokens representing robot motor commands (e.g., joint velocities/positions, gripper states).
Scoring recipe
def compute_success_rate(model, tasks, trials_per_task=20):
total_successes = 0
total_trials = 0
for task in tasks:
for config in task.predefined_layouts:
for trial in range(trials_per_task):
obs = get_observation(config, trial)
action = model.predict(obs)
success = check_objective_criteria(action, config)
total_successes += success
total_trials += 1
return (total_successes / total_trials) * 100
Common pitfalls
- Operator bias can skew real-robot results; the paper enforces a blind black-box inference server to prevent this.
- Success criteria must be strictly pre-defined and objective, as human operators record binary outcomes without knowing the model identity.
- LIBERO and RoboCasa use different trial counts (500 vs 50) and scene splits, so averaging must follow each benchmark's specific protocol.
Evidence (verbatim from paper)
We report the mean success rate over 50 evaluation episodes per task. Being-H0.5(specialist) reaches a 98.9% average success rate on LIBERO, with strong performance across all suites including LIBERO-Long (97.4%).
Citation
@misc{luo2026beingh05,
title={Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalization},
author={Hao Luo et al. (2026)},
year={2026},
note={arXiv:2601.12993}
}
- arXiv: 2601.12993