Mimicking Bench Eval

mimicking-bench-eval

Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimicking — Yun Liu et al. (2024) (arXiv:2412.17730, 2024)

What this evaluates

Evaluates generalizable humanoid-scene interaction learning by testing motion retargeting, tracking, and imitation learning across six household tasks. It probes the agent's ability to mimic human references to interact with diverse, unseen object geometries while maintaining physical plausibility and energy efficiency.

Datasets

• Mimicking-Bench — total 23000; splits: train (-1), test (-1)

Metrics

• kinematic success rate (primary) — range: percent
  • Percentage of task episodes where the humanoid's final pose and contact satisfy the kinematic metric K (e.g., correct body configuration and object interaction points).
• energy-averaged success rate — range: percent
  • Success rate combining kinematic metric K and physical metric P (energy consumption/stability) using a weighting factor λ_P. Reported as the second value in the 'K/Energy' format.

Input / output format

Input: Elevation map (E) of the scene as the sole visual observation, combined with proprioceptive state and task instructions.

Output: Continuous action commands (joint torques/positions) for the humanoid robot to execute the specified interaction task.

Scoring recipe

def compute_success_rates(predictions, gold_tasks, lambda_p_values):
    kinematic_success = 0
    energy_success = 0
    for pred, task in zip(predictions, gold_tasks):
        k_score = evaluate_kinematics(pred, task)
        p_score = evaluate_physics(pred, task)
        if k_score >= K_THRESHOLD:
            kinematic_success += 1
        if k_score >= K_THRESHOLD and p_score >= P_THRESHOLD(lambda_p_values):
            energy_success += 1
    return (kinematic_success / len(predictions)) * 100, \
           (energy_success / len(predictions)) * 100

Common pitfalls

• Treating kinematic success as sufficient for physical plausibility; the paper shows data-free RL can be kinematically successful but physically implausible (high energy/unstable).
• Assuming multi-view RGBD inputs are superior; the benchmark explicitly demonstrates that elevation maps significantly outperform RGBD due to egocentric noise.
• Ignoring the object-level 3:1 train-test split; generalization is evaluated on unseen object geometries, not just unseen tasks.

Evidence (verbatim from paper)

In the following tables, we report both the kinematic success rate, determined by the kinematic metric (K), and the energy-averaged success rate, which is judged by K and the physical metric P using four different λ_P values, as described in Section 3.2. These two success rates are separated by a forward slash.

Citation

@misc{liu2024mimickingbench,
  title={Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimicking},
  author={Yun Liu et al. (2024)},
  year={2024},
  note={arXiv:2412.17730}
}

• arXiv: 2412.17730