Jigsaw Puzzle Tiling Eval

jigsaw-puzzle-tiling-eval

DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation — Wan et al. (2020) (arXiv:2005.02588, 2020)

What this evaluates

Evaluates object detection, recognition, and precise tiling assembly capabilities for modular robotic manipulation. It tests the pipeline's ability to segment oddly shaped pieces, recognize their identity, and assemble them with high spatial accuracy.

Datasets

• Jigsaw puzzle set — total 4; splits: (unstated)

Metrics

• score (primary) — range: [0, 1]
  • Area rate defined as A_standard / A_actual, where A_standard is the area of perfectly aligned pieces and A_actual is the area of the minimum bounding box of the assembled pieces.
• IoU — range: [0, 1]
  • Intersection over Union for segmentation accuracy.
• AP — range: [0, 1]
  • Average Precision for object recognition accuracy.

Input / output format

Input: RGB image and depth map of randomly placed jigsaw pieces on a table.

Output: Bounding box, piece identity, grasp orientation, and placement coordinates.

Scoring recipe

def score_jigsaw(predictions, gold):
    A_standard = 10.2 * 10.2
    A_actual = compute_min_bounding_box_area(predictions)
    score = A_standard / A_actual
    iou = compute_iou(predictions, gold)
    ap = compute_ap(predictions, gold)
    return score, iou, ap

Common pitfalls

• Uses a suction cup instead of a standard parallel gripper due to the 5mm thickness of the pieces.
• Training data for segmentation was small (~80 images) and collected on a specific camera (D435), causing potential domain gaps on other sensors.

Evidence (verbatim from paper)

For performance assessment of the overall task, we report a 2D measurement of the area rate defined by score = A_standard / A_actual to evaluate the task completion... we report the IoU for segmentation, average precision (AP) for recognition, the success rate of picking and time cost of a pick and place sub-task

Citation

@misc{wan2020deepclaw,
  title={DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation},
  author={Wan et al. (2020)},
  year={2020},
  note={arXiv:2005.02588}
}

• arXiv: 2005.02588