Structured3d Layout Eval

structured3d-layout-eval

SpatialLM: Training Large Language Models for Structured Indoor Modeling — Mao et al. (2025) (arXiv:2506.07491, 2025)

What this evaluates

Evaluates a model's ability to predict architectural elements (walls, doors, windows) and room layouts within indoor 3D scenes. It tests the model's capacity for structured scene understanding and spatial reasoning by comparing predicted layouts against ground-truth annotations.

Datasets

• Structured3D — total 3500; splits: train (3000), val (250), test (250)

Metrics

• F1 (primary) — range: percent
  • F1 score computed using 2D Intersection over Union (IoU) between predicted and ground truth room layouts at a threshold of 0.25.

Input / output format

Input: 3D point clouds of indoor scenes.

Output: Structured 3D layouts describing walls, doors, and windows (polygons and corners).

Scoring recipe

def compute_f1(preds, golds, iou_thresh=0.25):
    tp, fp, fn = 0, 0, 0
    for pred, gold in zip(preds, golds):
        iou = compute_iou_2d(pred, gold)
        if iou >= iou_thresh:
            tp += 1
        else:
            fp += 1
    fn = len(golds) - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• RoomFormer converts 2D layouts to 3D by simple extrusion, which may not reflect true 3D geometry.
• Auto-regressive models do not output confidence scores, so standard mAP cannot be used; F1 is reported instead.

Evidence (verbatim from paper)

Following the setting of RoomFormer, we perform evaluation on the Structured3D benchmark, which contains 3,500 residential houses with diverse floorplans. We use the original data split of 3000/250/250 for training/validation/testing, respectively. Since auto-regressive models (i.e., SpatialLM and SceneScript) do not produce confidence scores, we report F1 scores instead of mean Average Precision (mAP), as suggested in[SceneScript].

Citation

@misc{mao2025spatiallm,
  title={SpatialLM: Training Large Language Models for Structured Indoor Modeling},
  author={Mao et al. (2025)},
  year={2025},
  note={arXiv:2506.07491}
}

• arXiv: 2506.07491