Jigsaw Robot Manipulation Eval

jigsaw-robot-manipulation-eval

Jigsaw-based Benchmarking for Learning Robotic Manipulation — Liu et al. (2023) (arXiv:2306.04932, 2023)

What this evaluates

Evaluates spatial-temporal reasoning and hardware-agnostic robotic manipulation skills through a structured jigsaw puzzle assembly protocol. It measures vision-based segmentation, object recognition, pick planning success, and motion planning efficiency across three progressively complex physical tasks.

Datasets

• Jigsaw Manipulation Benchmark — total ?; splits: test (-1)

Metrics

• IoU — range: [0, 1]
  • Intersection over Union between the predicted bounding box and the ground truth bounding box for segmentation. Calculated by comparing a background image with the input image containing jigsaw pieces.
• AP — range: [0, 1]
  • Average Precision for object classification, calculated as M/4 where M is the number of correct predictions for the four fragmented pieces. Calculated manually.
• Success rate — range: [0, 1]
  • Ratio of successful grasps to total attempts. In this specific setup using a suction cup on plate objects, it is always 1.
• Grasping time — range: seconds
  • Time in seconds required for the arm to execute the pick and place motion. Varies by arm and trajectory.
• Task Score (primary) — range: [0, 1]
  • Varies by task: Pick-and-place = correct placements / 4; Tiling = standard area / real area within bounding box; Assembly = correct placements / 4. Measures overall task completion.

Input / output format

Input: RGB/D images from a fixed camera (1m above table), robot joint states, and predefined workspace coordinates. The system receives visual input to segment jigsaw pieces, recognize their types, and plan grasps/placements.

Output: Robot control commands (joint trajectories, gripper actuation) and visual outputs (bounding boxes for segmentation, class labels for recognition). Task completion is reported as a scalar score and timing metrics.

Scoring recipe

def compute_task_score(task_type, predictions, gold):
    if task_type == 'pick_place':
        correct = sum(1 for p in predictions if p in gold['target_boxes'])
        return correct / 4
    elif task_type == 'tiling':
        return gold['standard_area'] / compute_bounding_box_area(predictions)
    elif task_type == 'assembly':
        correct = sum(1 for p in predictions if p in gold['base_plate_inner_box'])
        return correct / 4

def compute_iou(pred_box, gt_box):
    intersection = area_overlap(pred_box, gt_box)
    union = area_union(pred_box, gt_box)
    return intersection / union if union > 0 else 0

def compute_ap(predictions, ground_truth):
    correct = sum(1 for p in predictions if p in ground_truth)
    return correct / 4

Common pitfalls

• The success rate for pick planning is artificially high (always 1) due to the suction cup's effectiveness on plate objects, making it useless for differentiating between algorithms.
• The tiling task's area metric can be skewed by the bounding box calculation if pieces are not tightly packed or if occlusion occurs, leading to inconsistent spatial measurements.
• Hardware differences (arm workspace, camera resolution, calibration) significantly impact vision and motion planning metrics, making direct cross-platform comparison sensitive to setup variations.

Evidence (verbatim from paper)

IoU is the metric of the segmentation function, and it measures the positioning performance. To calculate the IoU, we first capture an image without jigsaw puzzles as a background and collect an image with jigsaw puzzles as input when the task begins. We find the difference between the background and input images as ground truth. Use the rectangle predicted and ground truth to calculate the IoU. AP is the metric of the recognition function; it measures the object classification accuracy. In each experiment, we count the number M of correct prediction, and AP is M/4 (as we only use four fragmented pieces), calculated manually. Success rate is the metric of the pick planning function; it measures the performance of grasping. As the suction cup is perfect for picking plate objects, the success rate is always 1, which means each grasping is successful. Score is the metric of the full task. It measures the performance of the entire task and is different for different tasks. For pick and place tasks, the score is the number of fragmented jigsaw pieces divided by 4, which are placed in the four squared areas on the A4-sized paper without overlap of the box. For the tiling task, the score

Citation

@misc{liu2023jigsawbenchmarking,
  title={Jigsaw-based Benchmarking for Learning Robotic Manipulation},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2306.04932}
}

• arXiv: 2306.04932