Surprise3d Eval

surprise3d-eval

SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes — Huang et al. (2025) (arXiv:2507.07781, 2025)

What this evaluates

This benchmark evaluates language-guided spatial understanding and reasoning in complex 3D scenes. It probes a model's ability to reason about relative positions, narrative/parametric perspectives, and absolute distances without relying on semantic shortcuts or explicit object names in the prompts.

Datasets

• SURPRISE3D — total 200000; splits: test (-1); repo https://github.com/liziwennba/SUPRISE

Metrics

• Accuracy (A25/A50) (primary) — range: [0, 100]
  • Binary accuracy computed at intersection-over-union thresholds of 0.25 (A25) and 0.50 (A50). Returns 1 (or 100) if IoU >= threshold, else 0.
• mIoU — range: [0, 100]
  • Mean Intersection over Union measuring the average overlap ratio between predicted and ground-truth 3D volumes.

Input / output format

Input: 3D scenes represented as point clouds, paired with images and natural language questions/prompts focusing on knowledge and spatial reasoning.

Output: 3D segmentation masks or bounding boxes identifying the target objects referenced in the prompt.

Scoring recipe

def compute_metrics(pred_mask, gt_mask):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = np.logical_or(pred_mask, gt_mask).sum()
    iou = intersection / union if union > 0 else 0.0
    miou = iou * 100
    a25 = 100.0 if iou >= 0.25 else 0.0
    a50 = 100.0 if iou >= 0.50 else 0.0
    return miou, a25, a50

Common pitfalls

• Models may exploit semantic shortcuts (e.g., object names in prompts) rather than true spatial reasoning; the dataset explicitly omits object names to force genuine reasoning.
• Confusion between evaluation protocols for segmentation models (which output masks and use mIoU) versus detection models (which output bounding boxes and use Accuracy thresholds).
• Assuming zero-shot results reflect inherent capability, as fine-tuning yields ~3x improvements, indicating prior models lack built-in spatial reasoning for these specific tasks.

Evidence (verbatim from paper)

For the segmentation tasks (MLLMfor3D and Reason3D), we adopt both Mean Intersection over Union (MIoU) and Accuracy (Acc) as evaluation metrics. MIoU measures the average overlap between the predicted and true 3D volumes, while Accuracy evaluates precision across varying confidence thresholds, which we obtain from different intersection proportions (e.g., 0.25 and 0.50) of the predicted and ground-truth volumes.

Citation

@misc{huang2025surprise3d,
  title={SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes},
  author={Huang et al. (2025)},
  year={2025},
  note={arXiv:2507.07781}
}

• arXiv: 2507.07781