Reconstruction Pose Eval

reconstruction-pose-eval

Comparative Evaluation of 3D Reconstruction Methods for Object Pose Estimation — Burde et al. (2024) (arXiv:2408.08234, 2024)

What this evaluates

Evaluates how well different 3D reconstruction methods perform in a downstream object pose estimation task, rather than measuring standalone geometric reconstruction accuracy. It compares pose estimation results using reconstructed 3D models against those using ground-truth CAD models.

Datasets

• YCB-V — total ?; splits: train (-1), test (-1); repo https://github.com/VarunBurde/reconstruction_pose_benchmark

Metrics

• accuracy of the estimated poses (primary) — range: percent
  • Measures the percentage of correctly estimated 6D object poses (rotation and translation) against ground-truth poses, following the standard evaluation guidelines of the BOP benchmark. Typically computed using distance/angle thresholds or ADD/ADI metrics.

Input / output format

Input: Undistorted multi-view images of an object, camera intrinsics and extrinsics, object masks, and a 3D mesh (either reconstructed or ground-truth CAD) to be registered and used for pose estimation.

Output: Estimated 6D pose (rotation matrix and translation vector) for each object in the test images, defined in the object's coordinate frame.

Scoring recipe

# Register reconstructed mesh to CAD frame via ICP
mesh = register_to_cad(reconstructed_mesh)
# Run pose estimator on test images
for img, gt_pose in test_set:
    pred_pose = estimator.predict(img, mesh, intrinsics)
    error = compute_bop_error(pred_pose, gt_pose)
    if error < threshold: correct += 1
accuracy = correct / len(test_set)

Common pitfalls

• Coordinate frame mismatch between reconstruction and CAD/test frames requires precise ICP registration before evaluation.
• Pose refinement must be disabled or standardized across methods to ensure all reconstructions share the same coordinate frame.
• Variable number of capture images per object (397–505) affects reconstruction quality and downstream pose accuracy.

Evidence (verbatim from paper)

Thus, we mainly measure 3D reconstruction performance by the accuracy of the estimated poses rather than the accuracy of the resulting 3D models itself. The evaluation of the pose estimators runs on the classic YCB-V pose estimation dataset and follows the standard guidelines defined in the BOP benchmark.

Citation

@misc{burde2024reconstructionposebenchmark,
  title={Comparative Evaluation of 3D Reconstruction Methods for Object Pose Estimation},
  author={Burde et al. (2024)},
  year={2024},
  note={arXiv:2408.08234}
}

• arXiv: 2408.08234