Bop 6d Pose Refinement Eval

bop-6d-pose-refinement-eval

GenFlow: Generalizable Recurrent Flow for 6D Pose Refinement of Novel Objects — Moon et al. (2024) (arXiv:2403.11510, 2024)

What this evaluates

Evaluates the ability to refine 6D object poses in cluttered real-world scenes using RGB or RGB-D inputs. It probes generalization to novel objects by measuring pose accuracy against ground truth under symmetry-aware error metrics.

Datasets

• LM-O — total ?; splits: test (-1)
• T-LESS — total ?; splits: test (-1)
• TUD-L — total ?; splits: test (-1)
• IC-BIN — total ?; splits: test (-1)
• ITODD — total ?; splits: test (-1)
• HomebrewdDB — total ?; splits: test (-1)
• YCB-V — total ?; splits: test (-1)

Metrics

• Average Recall (AR) (primary) — range: percent
  • AR is computed by averaging recall over multiple correctness thresholds for three pose-error functions: Visible Surface Discrepancy (VSD), Maximum Symmetry-Aware Surface Distance (MSSD), and Maximum Symmetry-Aware Projection Distance (MSPD). A pose is considered correct if any error function falls below its threshold.

Input / output format

Input: RGB or RGB-D images of cluttered real-world scenes containing multiple objects, along with 2D bounding box detections and an initial 6D pose hypothesis for each object.

Output: Refined 6D pose (rotation matrix and translation vector) for each detected object instance.

Scoring recipe

def compute_ar(predictions, ground_truths):
    correct = 0
    for pred, gt in zip(predictions, ground_truths):
        vsd_err = compute_vsd(pred.pose, gt.pose)
        mssd_err = compute_mssd(pred.pose, gt.pose)
        mspd_err = compute_mspd(pred.pose, gt.pose)
        if (vsd_err < VSD_TOL or mssd_err < MSSD_THRESH or mspd_err < MSPD_THRESH):
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Failing to match the multi-hypothesis strategy settings (n=10 hypotheses, 5 outer updates, 8 inner updates) when comparing against baselines like MegaPose.
• Confusing RGB-only and RGB-D input settings, which require different evaluation splits and yield different performance baselines.
• Not adhering to the standard BOP challenge evaluation protocol for pose error functions and threshold averaging.

Evidence (verbatim from paper)

Following the evaluation methodology of BOP challenge [[28]], we report the average recall (AR) considering three pose-error functions: Visible Surface Discrepancy (VSD), Maximum Symmetry-Aware Surface Distance (MSSD), and Maximum Symmetry-Aware Projection Distance (MSPD). In consideration of the global symmetry information, MSSD computes the maximum distance between the estimated camera-space coordinates and corresponding ground truth. In contrast, MSPD computes the maximum distance between the estimated image-space coordinates and the corresponding ground truth. Concerning each pose-error function, the AR is obtained by averaging recall calculated for multiple settings of the correctness thresholds (and misalignment tolerances for VSD).

Citation

@misc{moon2024genflow,
  title={GenFlow: Generalizable Recurrent Flow for 6D Pose Refinement of Novel Objects},
  author={Moon et al. (2024)},
  year={2024},
  note={arXiv:2403.11510}
}

• arXiv: 2403.11510