Maniskill2 Softbody Eval

maniskill2-softbody-eval

Learning the Generalizable Manipulation Skills on Soft-body Tasks via Guided Self-attention Behavior Cloning Policy — Li et al. (2024) (arXiv:2410.05756, 2024)

What this evaluates

Evaluates a robot policy's ability to perform long-horizon manipulation tasks involving deformable soft bodies (e.g., clay, noodles, liquid, plasticine). It probes spatial reasoning, contact dynamics, and precise end-effector control under varying initial conditions.

Datasets

• ManiSkill2 Challenge (Soft-body Track) — total ?; splits: test (450)

Metrics

• Success Metric (primary) — range: [0, 1]
  • Fraction of successful episodes out of total evaluation episodes per task. Success is determined by task-specific criteria: volume thresholds (Fill), geometric constraints (Hang, Excavate, Pour), Chamfer distance (Pinch), or IoU (Write), plus velocity limits (<0.05) where specified.

Input / output format

Input: Point cloud observations and end-effector state in the robot's frame, processed via a guided self-attention module for long-horizon interaction modeling.

Output: End-effector delta position commands via a pd-joint-delta-pos controller.

Scoring recipe

def compute_success_rate(task, episodes):
    successes = 0
    for ep in episodes:
        if task == 'Fill':
            ok = (ep.volume_in_beaker > 0.9 * ep.capacity) and (ep.velocity < 0.05)
        elif task == 'Hang':
            ok = (ep.noodle_above_rod) and (ep.ends_on_opposite_sides) and (not ep.touches_ground) and (ep.gripper_open) and (ep.velocity < 0.05)
        elif task == 'Excavate':
            ok = (ep.lifted_volume >= ep.threshold) and (ep.height >= ep.height_threshold) and (ep.spillage < 20) and (ep.velocity < 0.05)
        elif task == 'Pour':
            ok = (abs(ep.liquid_level - ep.target_level) <= 0.004) and (ep.spillage < 100) and (ep.bottle_upright) and (ep.arm_velocity < 0.05)
        elif task == 'Pinch':
            ok = (ep.chamfer_dist < 0.3 * ep.initial_chamfer_dist)
        elif task == 'Write':
            ok = (ep.iou > 0.8)
        else:
            ok = False
        if ok: successes += 1
    return successes / len(episodes)

Common pitfalls

• Task-specific success criteria vary widely (e.g., Chamfer distance vs. IoU vs. volume thresholds), requiring separate validation logic per task.
• Velocity constraints (<0.05) are mandatory for success in Fill, Hang, Excavate, and Pour, but omitted for Pinch and Write; ignoring them causes false positives.
• Evaluation uses randomized initial conditions (poses, target shapes/characters), making results highly sensitive to the random seed and episode count.

Evidence (verbatim from paper)

ManiSkill2 challenge includes 6 soft-body manipulation tasks that call for agents to engage with soft bodies (refer to Fig. 4), moving or deforming them to achieve predetermined target states. Success Metric: The task is successful when the volume of clay inside the target beaker exceeds 90% of its capacity, while maintaining the soft body velocity below 0.05. Evaluation Protocol: Conduct 100 episodes with varying initial rotations of the bucket and initial positions of the beaker.

Citation

@misc{li2024learninggeneralizable,
  title={Learning the Generalizable Manipulation Skills on Soft-body Tasks via Guided Self-attention Behavior Cloning Policy},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2410.05756}
}

• arXiv: 2410.05756