Graspclutter6d Eval

graspclutter6d-eval

GraspClutter6D: A Large-scale Real-world Dataset for Robust Perception and Grasping in Cluttered Scenes — Back et al. (2025) (arXiv:2504.06866, 2025)

What this evaluates

Evaluates robotic perception and manipulation capabilities in highly cluttered, real-world environments. It benchmarks instance segmentation, 6D object pose estimation, and 6-DoF grasp detection under varying levels of occlusion and scene complexity.

Datasets

• GraspClutter6D — total 1000; splits: train (413), test (235)

Metrics

• Grasp Success Rate (GSR) (primary) — range: percent
  • Ratio of successful lifts without slippage to total attempts.
• Declutter Rate (DR) — range: percent
  • Average ratio of objects cleared per round.
• COCO AP — range: [0, 1]
  • Average Precision for instance segmentation, computed using standard COCO evaluation metrics.
• ADD-S — range: [0, 1]
  • Area under the accuracy-threshold curve up to 10 cm for 6D pose estimation.
• APμ — range: [0, 1]
  • Average success rate of the top 50 predicted grasps across friction coefficients μ from 0.2 to 1.2 at 0.2 intervals, using the force-closure metric.

Input / output format

Input: RGB-D images or point clouds of cluttered scenes (table, shelf, bin) with varying object counts (5-15) and occlusion levels.

Output: Predicted 6-DoF grasps, segmentation masks, or 6D object poses.

Scoring recipe

def compute_metrics(predictions, gold):
    successful = sum(1 for g in predictions if g.lift_success)
    cleared = sum(1 for obj in gold if obj.cleared)
    GSR = successful / len(predictions)
    DR = cleared / len(gold)
    AP = coco_compute_ap(predictions.masks, gold.masks)
    ADD_S = compute_auc(predictions.poses, gold.poses, threshold=0.1)
    AP_mu = mean([compute_grasp_ap(predictions.grasps, mu) for mu in np.arange(0.2, 1.21, 0.2)])
    return GSR, DR, AP, ADD_S, AP_mu

Common pitfalls

• Models are evaluated only on foreground grasps (≤5cm from target objects) to remove background-induced ambiguity.
• Occlusion levels are explicitly stratified (low: v≥0.9, medium: 0.9>v≥0.6, high: 0.6>v) and must be reported separately.
• Grasp detection AP is averaged across friction coefficients (μ=0.2 to 1.2), not just a single value.

Evidence (verbatim from paper)

We compute the standard average precision (APμ), which measures the average success rate of the top 50 predicted grasps at friction coefficient μ using the force-closure metric. We report AP as a primary metric, averaging APμ across friction coefficients from 0.2 to 1.2 at 0.2 intervals.

Citation

@misc{back2025graspclutter6d,
  title={GraspClutter6D: A Large-scale Real-world Dataset for Robust Perception and Grasping in Cluttered Scenes},
  author={Back et al. (2025)},
  year={2025},
  note={arXiv:2504.06866}
}

• arXiv: 2504.06866