Object Pose Estimation Robotics Eval

object-pose-estimation-robotics-eval

Object Pose Estimation in Robotics Revisited — Hietanen et al. (2019) (arXiv:1906.02783, 2019)

What this evaluates

This benchmark evaluates 6D object pose estimation for robotic manipulation tasks. It probes whether estimated poses are sufficiently accurate to enable successful physical assembly or grasping, rather than just measuring geometric alignment.

Datasets

• Industrial Object Pose Dataset — total 600; splits: test (600)

Metrics

• Average success probability (primary) — range: [0, 1]
  • The probability of task success P(X=1|θ̂) estimated via non-parametric kernel regression on real-world robot trial data, evaluated at the residual pose from the canonical grasp pose.
• % p ≥ 0.9 — range: percent
  • The proportion of test images for which the estimated success probability is greater than or equal to 0.90.
• ADC error — range: other
  • Average Distance of Model Points: (1/|M|) Σ ||Ŷx - Υx||, where M is the set of model 3D points, Ŷ is the estimated pose, and Υ is the ground truth pose. Also reported as top-25% ADC error to reduce outlier sensitivity.

Input / output format

Input: Downsampled point clouds of the object model and scene (RGB-D data), with surface normals estimated via least squares plane fitting. Voxels sized 0.5–1.0 mm. Local descriptors (SHOT) computed on 1000–3000 uniformly sampled points.

Output: 6D object pose estimate relative to the sensor, represented as a 4×4 transformation matrix or 6D vector.

Scoring recipe

def compute_metrics(pred_poses, gt_poses, model_points):
    success_probs = []
    adc_errors = []
    for pred, gt in zip(pred_poses, gt_poses):
        residual = pose_residual(pred, gt)
        p = kernel_model.predict(residual)
        success_probs.append(p)
        pred_pts = transform(model_points, pred)
        gt_pts = transform(model_points, gt)
        adc = mean_l2_distance(pred_pts, gt_pts)
        adc_errors.append(adc)
    avg_sp = mean(success_probs)
    pct_ge_09 = sum(1 for p in success_probs if p >= 0.9) / len(success_probs)
    top25_adc = percentile(adc_errors, 25)
    return avg_sp, pct_ge_09, adc_errors, top25_adc

Common pitfalls

• ADC error degrades linearly and is uninformative for task success; it does not capture the sharp threshold where physical success drops to zero.
• Coordinate systems must be correctly aligned: the pose estimate must be converted to the object-relative canonical grasp frame before evaluating success probability.
• The success probability metric requires a pre-computed kernel regression model trained on physical robot trials; it cannot be computed from geometry alone.

Evidence (verbatim from paper)

The main performance metric in our work is the estimated success probability defined in Section 3.1. The probabilities were computed around the canonical grasp pose of each object and therefore the sampled values actually represent residual from this pose. We calculated the average probabilities over the whole dataset and also the proportion of images for which the probability is greater or equal to 0.90. In addition to the proposed indicator we also report the ADC error calculated over the points transformed by the ground truth and estimated object pose as suggested in[11].

Citation

@misc{hietanen2019objectpose,
  title={Object Pose Estimation in Robotics Revisited},
  author={Hietanen et al. (2019)},
  year={2019},
  note={arXiv:1906.02783}
}

• arXiv: 1906.02783