Maniskill Hab Eval

maniskill-hab-eval

ManiSkill-HAB: A Benchmark for Low-Level Manipulation in Home Rearrangement Tasks — Shukla et al. (2024) (arXiv:2412.13211, 2024)

What this evaluates

Evaluates low-level robotic manipulation policies for long-horizon home rearrangement tasks. It probes a robot's ability to successfully pick, place, and interact with household objects across cluttered and constrained environments.

Datasets

• ManiSkill-HAB — total ?; splits: train (-1), val (-1)

Metrics

• success once rate (primary) — range: percent
  • Percentage of episodes where a specific subtask (e.g., Pick, Place, Open/Close) is completed successfully on the first attempt. Computed per subtask across train and validation splits.
• progressive completion rate — range: percent
  • Percentage of long-horizon episodes where the entire sequence of subtasks is completed successfully. An optimistic upper bound is calculated by multiplying individual subtask success rates.
• failure mode ratio — range: ratio
  • Ratio of 'place in goal' to 'drop to goal' trajectories for imitation learning policies, used to measure behavioral control after trajectory filtering.

Input / output format

Input: Simulated robot state and observations (e.g., RGB-D frames) in a home environment with target objects and receptacles.

Output: Low-level continuous control actions for the robot arm and base.

Scoring recipe

def compute_metrics(predictions, gold, episodes):
    subtask_success = []
    for ep in episodes:
        subtask_success.append(1.0 if ep.completed_subtask else 0.0)
    success_once_rate = sum(subtask_success) / len(subtask_success) * 100
    progressive_completion = sum(1.0 for ep in episodes if ep.completed_all_subtasks) / len(episodes) * 100
    upper_bound = 1.0
    for subtask in subtasks:
        upper_bound *= compute_metrics(predictions, gold, subtask)
    return success_once_rate, progressive_completion, upper_bound

Common pitfalls

• Assuming subtask completions are independent when calculating the optimistic upper bound, which overestimates long-horizon success.
• Ignoring scene geometry constraints in the validation split (e.g., fridge doors opening into walls) that cause policy failure despite good training performance.
• Overlooking handoff issues or disturbances to previously placed objects when evaluating multi-step tasks.

Evidence (verbatim from paper)

Fig. [4] shows the RL and IL policies’ progressive completion rate. We provide an optimistic upper bound on progressive completion rate by (incorrectly) assuming that the completion of each subtask is independent of every other subtask, thus directly multiplying subtask success once rates. Table [1] shows success once rate for individual subtasks.

Citation

@misc{shukla2024maniskillhab,
  title={ManiSkill-HAB: A Benchmark for Low-Level Manipulation in Home Rearrangement Tasks},
  author={Shukla et al. (2024)},
  year={2024},
  note={arXiv:2412.13211}
}

• arXiv: 2412.13211