Claw Machine Bin Clearing Eval

claw-machine-bin-clearing-eval

DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation — Wan et al. (2020) (arXiv:2005.02588, 2020)

What this evaluates

Measures end-to-end robotic manipulation performance and grasp robustness by clearing a bin of soft objects using a learned policy. It evaluates the ability to predict optimal grasping poses from RGB-D inputs and execute them across different hardware platforms.

Datasets

• Soft toy bin-clearing set — total 8; splits: (unstated)

Metrics

• r_success (primary) — range: [0, 1]
  • Total picking success rate, calculated as 8 divided by the total number of grasp attempts required to clear the bin.
• t_pick — range: seconds
  • Robot arm execution time for a single pick task, excluding gripper closing time. Averaged over 10 runs.

Input / output format

Input: RGB image and depth map of the bin containing soft toys.

Output: Grasp pose (x, y, z) and orientation (theta) for vertical grasps.

Scoring recipe

def score_claw(predictions, gold):
    attempts = count_grasp_attempts(predictions)
    success_rate = 8.0 / attempts
    pick_times = [measure_time(p) - gripper_close_time for p in predictions]
    avg_time = sum(pick_times) / len(pick_times)
    return success_rate, avg_time

Common pitfalls

• Gripper closing time varies significantly across hardware (e.g., 0.80s vs 2.0s) and must be subtracted from t_pick.
• Grasps are strictly restricted to vertical orientation with only z-axis rotation allowed; non-compliant poses invalidate the attempt.

Evidence (verbatim from paper)

For performance assessment, we report robot arm execution time $t_{pick}$ for a single pick task, and the total picking success rate $r_{success}$ , which equals to eight over the number of total grasp attempts before clearing the bin. We performed the bin-clearing task on three robot cells and ten times on each robot cell.

Citation

@misc{wan2020deepclaw,
  title={DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation},
  author={Wan et al. (2020)},
  year={2020},
  note={arXiv:2005.02588}
}

• arXiv: 2005.02588