Stanford 2d 3d S Eval

stanford-2d-3d-s-eval

INSIGHT: Indoor Scene Intelligence from Geometric-Semantic Hierarchy Transfer for Public~Safety — Dimopoulos et al. (2026) (arXiv:2604.23095, 2026)

What this evaluates

Evaluates 2D-to-3D semantic transfer pipelines for indoor scene understanding, focusing on per-point labeling accuracy for structural and furniture classes, and detection sensitivity for novel safety-critical objects in public safety contexts.

Datasets

• Stanford 2D-3D-S* — total ?; splits: test (-1)

Metrics

• per-point accuracy (primary) — range: [0, 1]
  • For each pipeline point whose semantic label maps to one of the 7 overlapping classes, the nearest Stanford GT point (by Euclidean distance in 3D) supplies the reference label. A point is correct if the two mapped labels agree. Per-area accuracy is a_k = n_k^correct / n_k^total, and overall accuracy is the area-weighted mean A = sum_k w_k a_k, where w_k = n_k^total / sum_j n_j^total.
• detection sensitivity — range: other
  • Raw count of detected instances per novel safety-critical class across all evaluated areas. No ground truth exists; reported as detection volume to compare pipeline performance.
• fragmentation ratio — range: other
  • Pipeline instances divided by reference count (GT instances for overlapping classes, code-derived plausibility cap K for novel classes) at a merge distance of 0.5 m.

Input / output format

Input: RGB+OpenEXR images and corresponding 3D point clouds from indoor scenes; pipeline receives 2D images to generate semantic labels and 3D points, which are then evaluated against Stanford 2D-3D-S ground truth point clouds.

Output: Per-point semantic labels mapped to a unified taxonomy (7 overlapping classes + 15 novel safety classes); detection counts for novel classes; instance coordinates for overlap and fragmentation analysis.

Scoring recipe

def compute_per_point_accuracy(pred_points, gt_points, valid_classes):
    correct = 0
    total = 0
    for p in pred_points:
        if p.label not in valid_classes:
            continue
        gt_label = find_nearest_gt_label(p.coords, gt_points, metric='euclidean')
        if p.label == gt_label:
            correct += 1
        total += 1
    area_acc = correct / total if total > 0 else 0.0
    return area_acc

Common pitfalls

• Structural classes (ceiling, floor, wall) show artificially low per-point accuracy due to single-instance-per-area compression and boundary overlaps with adjacent GT labels, not spatial misalignment.
• Novel safety-class detection counts are raw volumes without ground truth verification; they serve as proxy signals rather than strict precision/recall metrics.
• Inter-pipeline agreement is measured at a fixed 1 m centroid radius, which may over- or under-estimate true overlap for elongated or fragmented objects.

Evidence (verbatim from paper)

Per-point accuracy is computed over the 7 classes shared between the pipeline taxonomy and Stanford 2D-3D-S after class mapping (table+chair+sofa+bookcase → furniture; beam → column; six classes map directly). For each pipeline point whose semantic label maps to one of these 7 classes, the nearest Stanford GT point (by Euclidean distance in 3D) supplies the reference label. A point is correct if the two mapped labels agree. Points carrying pipeline-only labels (the 15 novel safety classes) are excluded; Stanford-only classes (board, clutter) have no pipeline equivalent and are excluded from the denominator. Per-area accuracy is $a_{k}=n_{k}^{\text{correct}}/n_{k}^{\text{total}}$ and the reported overall accuracy is the area-weighted mean $A=\sum_{k}w_{k},a_{k}$, where $w_{k}=n_{k}^{\text{total}}\big/\sum_{j}n_{j}^{\text{total}}$.

Citation

@misc{dimopoulos2026insight,
  title={INSIGHT: Indoor Scene Intelligence from Geometric-Semantic Hierarchy Transfer for Public~Safety},
  author={Dimopoulos et al. (2026)},
  year={2026},
  note={arXiv:2604.23095}
}

• arXiv: 2604.23095