Phocal Eval

phocal-eval

PhoCaL: A Multi-Modal Dataset for Category-Level Object Pose Estimation with Photometrically Challenging Objects — Wang et al. (2022) (arXiv:2205.08811, 2022)

What this evaluates

Evaluates category-level 6D object pose estimation on photometrically challenging objects (reflective, transparent, occluded). It tests both in-distribution generalization (seen objects) and out-of-distribution generalization (novel objects within the same category), comparing RGB-D and monocular approaches.

Datasets

• PhoCaL — total 24; splits: train (-1), test (-1)

Metrics

• 3D IoU (primary) — range: percent
  • Intersection over Union (IoU) between predicted and ground-truth 3D bounding boxes. Mean Average Precision (mAP) is computed at IoU thresholds of 25% and 50% across all categories and images.

Input / output format

Input: RGB or RGB-D image sequences containing household objects under varying illumination, occlusion, and partial visibility.

Output: 6D object pose (rotation and translation), scale, and 3D bounding box or point cloud shape.

Scoring recipe

def compute_3d_iou(pred_box, gt_box):
    intersection = volume_of_intersection(pred_box, gt_box)
    union = volume_of_union(pred_box, gt_box)
    return intersection / union if union > 0 else 0.0

def compute_mAP(predictions, ground_truths, iou_thresh=0.25):
    matches = []
    for pred, gt in zip(predictions, ground_truths):
        if compute_3d_iou(pred, gt) >= iou_thresh:
            matches.append(1.0)
        else:
            matches.append(0.0)
    return sum(matches) / len(matches) * 100

Common pitfalls

• Depth sensor artifacts on reflective/transparent objects cause significant performance drops for RGB-D methods, independent of the pose algorithm itself.
• Monocular methods suffer severe performance degradation on novel objects without extensive synthetic pretraining.
• Deformable objects (e.g., empty boxes) are difficult to annotate accurately and are not well-represented in the benchmark.

Evidence (verbatim from paper)

The evaluation metric is the intersection over union (IoU) result with a threshold of 25% and 50%. The mean average precision (mAP) for 3D IoU at 25% is 43.34%.

Citation

@misc{wang2022phocal,
  title={PhoCaL: A Multi-Modal Dataset for Category-Level Object Pose Estimation with Photometrically Challenging Objects},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2205.08811}
}

• arXiv: 2205.08811