Grasp Success Eval

grasp-success-eval

Benchmarking the Effects of Object Pose Estimation and Reconstruction on Robotic Grasping Success — Burde et al. (2026) (arXiv:2602.17101, 2026)

What this evaluates

This benchmark evaluates the functional impact of 6D object pose estimation and 3D mesh reconstruction methods on robotic grasping performance. It measures how geometric inaccuracies and spatial pose errors propagate to affect the success rate of physics-based grasping attempts in simulation.

Datasets

• YCB-Video (YCB-V) — total ?; splits: test (-1)

Metrics

• grasping success (primary) — range: percent
  • The proportion of attempted grasps that result in a successful physical interaction, calculated as (number of successful attempts) / (total number of attempts) per object and method combination.

Input / output format

Input: Estimated 6D object poses from MegaPose or FoundationPose, paired with reconstructed 3D meshes from various methods (NeRF, implicit surfaces, photogrammetry), for each instance in the YCB-V test set.

Output: Binary grasp outcome (success/failure) per simulation run, aggregated into a success rate.

Scoring recipe

success_count = 0
total_count = 0
for obj in dataset.test:
    for pose_est in [MegaPose, FoundationPose]:
        for recon in [Instant_NGP, NeRFacto, Neuralangelo, UniSurf, MonoSDF, BakedSDF, VolSDF, RealityCapture]:
            pose = pose_est(obj)
            mesh = recon(obj)
            for grasp in generate_grasps(mesh, pose):
                if physics_simulate(grasp):
                    success_count += 1
                total_count += 1
return success_count / total_count

Common pitfalls

• Geometric fidelity metrics (e.g., Chamfer distance) often fail to correlate with actual functional manipulation performance.
• For symmetric objects, spatial translation errors in pose estimation dominate grasp success, while mesh reconstruction quality primarily affects the quantity of viable grasp candidates rather than success rate when pose is accurate.

Evidence (verbatim from paper)

This paper introduces a physics-based benchmark that evaluates 6D pose estimation and 3D reconstruction methods by measuring their functional impact on robotic grasping success in a simulator. It reveals that geometric inaccuracies in reconstructed meshes significantly reduce the number of viable grasp candidates but have minimal effect on success when pose estimation is accurate, while spatial pose errors—especially translation—dominate grasp success for symmetric objects, exposing a critical disconnect between geometric fidelity metrics and real-world manipulation performance.

Citation

@misc{burde2026benchmarking,
  title={Benchmarking the Effects of Object Pose Estimation and Reconstruction on Robotic Grasping Success},
  author={Burde et al. (2026)},
  year={2026},
  note={arXiv:2602.17101}
}

• arXiv: 2602.17101