Industryshapes Eval

industryshapes-eval

IndustryShapes: An RGB-D Benchmark dataset for 6D object pose estimation of industrial assembly components and tools — Sapoutzoglou et al. (2026) (arXiv:2602.05555, 2026)

What this evaluates

This benchmark evaluates 6D object pose estimation, detection, and segmentation capabilities in realistic industrial environments. It specifically probes a model's ability to handle challenging conditions such as heavy occlusion, background clutter, reflective surfaces, textureless materials, and object symmetry.

Datasets

• IndustryShapes Classic — total 4600; splits: test (-1)
• IndustryShapes Extended — total ?; splits: test (-1)

Metrics

• Average Recall (AR) (primary) — range: [0, 1]
  • Mean recall across VSD, MSSD, and MSPD metrics. Recall is computed by counting predictions with error below a standard threshold (typically 2cm or 5°) divided by total ground-truth objects.
• VSD — range: [0, 1]
  • Visible Surface Discrepancy: computes the difference between renderings of the estimated and ground-truth poses only over the visible surface areas.
• MSSD — range: [0, 1]
  • Maximum Symmetry-Aware Surface Distance: measures the maximum surface distance between the estimated and ground-truth poses, accounting for object symmetry.
• MSPD — range: [0, 1]
  • Maximum Symmetry-Aware Projection Distance: measures the maximum 2D projection error in pixels between estimated and ground-truth poses, accounting for symmetry.
• ADD — range: [0, 1]
  • Average Distance for distinguishable objects: quantifies the average misalignment between the model’s vertices in the true and estimated pose.
• mAP — range: [0, 1]
  • mean Average Precision for object detection and segmentation, following standard BOP challenge conventions.

Input / output format

Input: RGB-D image frames containing industrial assembly components or tools, often with occlusions, clutter, and varying lighting. Ground-truth CAD models and initial pose estimates (for some methods) are provided for evaluation.

Output: Estimated 6D pose (3D rotation matrix/quaternion and 3D translation vector) per object, along with bounding boxes and segmentation masks for detection/segmentation tasks.

Scoring recipe

def compute_metrics(predictions, ground_truth):
    recalls = []
    for pred, gt in zip(predictions, ground_truth):
        vsd_err = compute_vsd(pred.pose, gt.pose, gt.visible_mask)
        mssd_err = compute_mssd(pred.pose, gt.pose, gt.symmetry)
        mspd_err = compute_mspd(pred.pose, gt.pose, gt.symmetry)
        # Standard BOP thresholds (e.g., 2cm/5deg)
        recalls.append(1 if vsd_err < 0.02 else 0)
        recalls.append(1 if mssd_err < 0.02 else 0)
        recalls.append(1 if mspd_err < 0.02 else 0)
    ar = sum(recalls) / len(recalls)
    mAP_det = compute_map(pred.bboxes, gt.bboxes)
    mAP_seg = compute_map(pred.masks, gt.masks)
    return {'AR': ar, 'mAP_det': mAP_det, 'mAP_seg': mAP_seg}

Common pitfalls

• Failing to account for object symmetry when computing surface or projection distances, which artificially inflates error for symmetric industrial parts.
• Confusing instance-level evaluation (trained per object) with novel-object evaluation (zero-shot/generalized), which require different baseline comparisons and training protocols.
• Using full-surface metrics (ADD, MSSD) on partially occluded objects without masking visible regions, leading to misleadingly high error rates compared to VSD.

Evidence (verbatim from paper)

We adhere to the BOP challenge protocol*, evaluating methods using four pose error metrics based on the estimated pose and the ground-truth pose: the Visible Surface Discrepancy (VSD), computing differences from renderings of the estimated and ground-truth poses only over the visible surface areas; the Maximum Symmetry-Aware Surface Distance (MSSD), measuring the maximum surface distance considering object symmetry; the Maximum Symmetry-Aware Projection Distance (MSPD) measuring the maximum 2D projection error in pixels, considering symmetry; and the Average Distance for distinguishable (ADD) objects which quantifies the average misalignment between the model’s vertices in the true and estimated pose. In line with the BOP protocol, the Average Recall (AR), used to summarize the overall performance, is computed as the mean recall of VSD, MSSD, and MSPD.

Citation

@misc{sapoutzoglou2026industryshapes,
  title={IndustryShapes: An RGB-D Benchmark dataset for 6D object pose estimation of industrial assembly components and tools},
  author={Sapoutzoglou et al. (2026)},
  year={2026},
  note={arXiv:2602.05555}
}

• arXiv: 2602.05555