Hummi Wholebody Manipulation Eval

hummi-wholebody-manipulation-eval

Humanoid Manipulation Interface: Humanoid Whole-Body Manipulation from Robot-Free Demonstrations — Nai et al. (2026) (arXiv:2602.06643, 2026)

What this evaluates

Evaluates a humanoid robot's capability to learn and execute whole-body manipulation skills from robot-free demonstrations, assessing manipulation precision, dynamic coordination, generalization to unseen environments/objects, and data-collection efficiency.

Datasets

• Squatting — total ?; splits: test (-1)
• Tossing — total ?; splits: test (-1)
• Bimanual Actions — total ?; splits: test (-1)
• Walking — total ?; splits: test (-1)
• Dynamic Motions — total ?; splits: test (-1)

Metrics

• success rate (primary) — range: percent
  • Percentage of trials where the robot successfully completes the manipulation task without failure, evaluated under in-domain and out-of-domain settings.
• tracking error — range: other
  • Mean Euclidean distance between the commanded trajectory and the executed trajectory, reported in centimeters.
• data-collection efficiency — range: other
  • Ratio of data points collected per unit time by HuMI compared to the baseline teleoperation system (TWIST2).

Input / output format

Input: Full-body motion trajectories captured via handheld sensors and wearable trackers, including gripper, base, and foot positions/orientations.

Output: Robot execution of the learned whole-body manipulation policy; success/failure outcome and trajectory tracking metrics.

Scoring recipe

def score(predictions, gold):
    success_count = sum(1 for p in predictions if p == 'success')
    success_rate = (success_count / len(predictions)) * 100
    tracking_error = mean(sqrt(sum((cmd - exec)^2, axis=1)))
    efficiency = data_collected_hu / data_collected_twist
    return success_rate, tracking_error, efficiency

Common pitfalls

• Evaluation covers both in-domain and out-of-domain settings, which significantly impacts reported success rates and generalization metrics.
• Data-collection efficiency is strictly compared against the TWIST2 teleoperation system, limiting direct comparison with other data collection methods.
• The exact threshold for 'successful' manipulation or acceptable tracking error is not specified in the provided text.

Evidence (verbatim from paper)

In this section, we empirically evaluate HuMI along three key dimensions. Specifically, we aim to answer the following questions: 1. Whole-body manipulation capability. Can HuMI learn feasible whole-body skills from robot-free demonstrations, achieve sufficient manipulation precision while respecting whole-body dynamics, and effectively coordinate high-low level policy during fully autonomous execution? 2. Generalization ability. Do robot-free demonstrations collected across varied environments enable the learned policy to generalize to unseen environments and objects? 3. Data-collection efficiency. Can HuMI acquire whole-body manipulation data efficiently and with a high acceptance rate, and does the collected dataset cover versatile whole-body skills, including motions that are challenging to obtain for teleoperation? To study these questions, we design five representative whole-body manipulation tasks and evaluate HuMI under both in-domain and out-of-domain settings with respect to environments and objects. To assess data-collection efficiency, we further compare HuMI against the state-of-the-art humanoid teleoperation system TWIST2.

Citation

@misc{nai2026hummi,
  title={Humanoid Manipulation Interface: Humanoid Whole-Body Manipulation from Robot-Free Demonstrations},
  author={Nai et al. (2026)},
  year={2026},
  note={arXiv:2602.06643}
}

• arXiv: 2602.06643