Mixed Bins Bench Eval

mixed-bins-bench-eval

Precise Object Placement with Pose Distance Estimations for Different Objects and Grippers — Kleeberger et al. (2021) (arXiv:2110.00992, 2021)

What this evaluates

Evaluates a robotic bin-picking framework's ability to estimate 6D object poses, select grasp candidates across parallel jaw and suction grippers, and precisely place objects in cluttered, symmetric, or entangled scenarios.

Datasets

• Mixed bins symmetries — total 500; splits: test (500)
• Mixed bins entanglements — total 500; splits: test (500)

Metrics

• AP average (primary) — range: [0, 1]
  • Average Precision for 6D object pose estimation, computed independently per object class and averaged. Based on the metric from Brégier et al.
• success rate grasping — range: [0, 1]
  • Proportion of trials where an object remains in the gripper after moving out of the bin.
• success rate object placement — range: [0, 1]
  • Proportion of trials where the placed object falls within a distance threshold of 0.1 times the object diameter from the target pose.
• avg. relative pose distance object placement — range: other
  • Mean relative pose distance between the actual and target placement poses across all test samples.

Input / output format

Input: Depth image of a cluttered bin containing multiple objects of varying shapes, sizes, and symmetries.

Output: 6D object pose estimates, object class predictions, grasp pose candidates (for parallel jaw and suction grippers), and pose distance estimations for placement confidence and precision.

Scoring recipe

# AP average
ap_scores = []
for cls in classes:
    preds = get_preds(cls)
    gold = get_golds(cls)
    ap_scores.append(compute_AP(preds, gold, metric=Brégier_et_al))
ap_avg = mean(ap_scores)

# Success rates
grasp_sr = sum(1 for t in trials if object_in_gripper(t)) / len(trials)
place_sr = sum(1 for t in trials if pose_dist(t.actual, t.target) <= 0.1 * t.diameter) / len(trials)

# Avg relative pose distance
avg_rel_dist = mean([pose_dist(t.actual, t.target) for t in trials])

Common pitfalls

• Baselines (OP-Net, PQ-Net) were extended by the authors to handle multiple object classes and grippers, meaning reported numbers may not reflect their original intended evaluation protocols.
• Dex-Net 4.0 is restricted to 4D (top-down) grasps for parallel jaw grippers, making direct comparison with the proposed 6D multi-gripper approach potentially unfair.
• Placement success uses a relative threshold (0.1 × object diameter), so absolute placement accuracy varies significantly across objects of different sizes.

Evidence (verbatim from paper)

All performance numbers are recorded on test datasets of 10 cycles each (500 samples). Table I reports the AP values based on the metric from Brégier et al. for the two challenging mixed bins datasets. Table II reports the success rates for grasping in simulation. A grasp is considered as successful if an object is in the gripper after moving out of the bin. Table III reports the results of our object placement experiments in simulation. The object placement success rate identifies whether the placement is within the distance threshold for all samples, where both approaches perform very similar. In our experiments, we log the relative pose distance for object placement for each trial and average it over all samples from the test dataset.

Citation

@misc{kleeberger2021precise,
  title={Precise Object Placement with Pose Distance Estimations for Different Objects and Grippers},
  author={Kleeberger et al. (2021)},
  year={2021},
  note={arXiv:2110.00992}
}

• arXiv: 2110.00992