Sole 3d Instance Segmentation Eval

sole-3d-instance-segmentation-eval

Segment Any 3D Object with Language — Lee et al. (2024) (arXiv:2404.02157, 2024)

What this evaluates

This benchmark evaluates a model's ability to perform open-vocabulary 3D instance segmentation on indoor point clouds. It probes the model's capacity to align 3D geometric features with free-form language instructions to generate accurate instance masks for both seen and unseen categories.

Datasets

• ScanNetv2 — total ?; splits: train (-1), val (-1), test (-1)
• ScanNet200 — total ?; splits: train (-1), val (-1), test (-1)
• Replica — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• AP (primary) — range: percent
  • Average Precision computed across IoU thresholds from 0.50 to 0.95 in steps of 0.05, following standard instance segmentation protocols. Also reported at fixed thresholds AP@25 and AP@50.

Input / output format

Input: 3D point cloud data (optionally augmented with projected 2D CLIP features) paired with free-form natural language instructions describing target objects.

Output: Binary or multi-instance segmentation masks over the 3D point cloud, corresponding to the objects specified in the language instruction.

Scoring recipe

def compute_ap(pred_masks, gt_masks, iou_thresholds):
    ap_scores = []
    for iou_thr in iou_thresholds:
        tp = sum(1 for p, g in zip(pred_masks, gt_masks) if calculate_iou(p, g) >= iou_thr)
        fp = len(pred_masks) - tp
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        ap_scores.append(precision)
    return np.mean(ap_scores) * 100

Common pitfalls

• Failing to exclude the 'other furniture' class when evaluating on ScanNetv2, which inflates results.
• Confusing the hierarchical open-set split (base vs. novel classes) with standard closed-set evaluation, leading to incorrect reporting of novel-category performance.
• Overlooking voxel size variations (2cm vs 4cm) which significantly impact memory, speed, and AP scores.

Evidence (verbatim from paper)

Average precision (AP) of different IoU thresholds is adopted as the evaluation metric, including AP under 25%, 50% IoU and the average AP from 50% to 95% IoU.

Citation

@misc{lee2024segmentany3d,
  title={Segment Any 3D Object with Language},
  author={Lee et al. (2024)},
  year={2024},
  note={arXiv:2404.02157}
}

• arXiv: 2404.02157