Openscan Eval

openscan-eval

OpenScan: A Benchmark for Generalized Open-Vocabulary 3D Scene Understanding — Zhao et al. (2024) (arXiv:2408.11030, 2024)

What this evaluates

Evaluates 3D vision models' ability to perform open-vocabulary scene understanding by querying for fine-grained object attributes (e.g., material, affordance, synonym) rather than standard object classes. It measures both 3D instance segmentation and 3D semantic segmentation capabilities on attribute-based queries across eight linguistic aspects.

Datasets

• OpenScan — total ?; splits: validation (-1)

Metrics

• AP (primary) — range: percent
  • Average Precision computed across multiple IoU thresholds (typically 0.25 to 0.95), averaged over all queried attributes.
• AP50 — range: percent
  • Average Precision calculated at a single IoU threshold of 0.50.
• AP25 — range: percent
  • Average Precision calculated at a single IoU threshold of 0.25.
• mIoU (primary) — range: percent
  • Mean Intersection over Union across all queried attributes, computed as the average of IoU scores for each attribute.
• mAcc — range: percent
  • Mean Accuracy across all queried attributes, representing the average proportion of correctly classified points/voxels per attribute.

Input / output format

Input: 3D scene data (point cloud or mesh) with aligned 2D RGB images, paired with a natural language query specifying an object attribute (e.g., 'made of wood', 'keep food cold').

Output: 3D instance masks or per-point/voxel semantic labels identifying objects in the scene that match the queried attribute.

Scoring recipe

def compute_ap(preds, gts, iou_thresh):
    matches = []
    for gt in gts:
        best_iou = max(iou(p, gt) for p in preds)
        matches.append(1.0 if best_iou >= iou_thresh else 0.0)
    return average_precision(matches)

def compute_miou(preds, gts):
    return sum(iou(p, gt) for p, gt in zip(preds, gts)) / len(gts)

Common pitfalls

• Models are evaluated strictly zero-shot; fine-tuning on OpenScan is prohibited.
• Query formulation drastically affects results: full sentence templates yield significantly higher scores than single keywords due to VLM limitations.
• Performance varies widely across the eight linguistic aspects; reporting only the mean masks critical capability gaps.

Evidence (verbatim from paper)

We evaluate OpenScene(Peng et al. [2023]), PLA(Ding et al. [2023]), and RegionPLC(Yang et al. [2024]), reporting the average score of all attributes in our OpenScan and that of all object classes in ScanNet(Dai et al. [2017]). Table[4] shows that although these OV-3D models perform well in recognizing object classes, they exhibit poor performances on linguistic aspects with low mIoU and mAcc metrics.

Citation

@misc{zhao2024openscan,
  title={OpenScan: A Benchmark for Generalized Open-Vocabulary 3D Scene Understanding},
  author={Zhao et al. (2024)},
  year={2024},
  note={arXiv:2408.11030}
}

• arXiv: 2408.11030