Panoptic Symbol Spotting Eval

panoptic-symbol-spotting-eval

ArchCAD-400K: An Open Large-Scale Architectural CAD Dataset and New Baseline for Panoptic Symbol Spotting — Luo et al. (2025) (arXiv:2503.22346, 2025)

What this evaluates

Evaluates a model's ability to simultaneously detect and classify architectural symbols in CAD drawings, distinguishing between discrete instances (things) and continuous background regions (stuff). It measures both geometric segmentation accuracy and semantic recognition quality.

Datasets

• ArchCAD-400K — total 413062; splits: train (289144), val (41306), test (82612)

Metrics

• Panoptic Quality (PQ) (primary) — range: [0, 1]
  • Combines segmentation quality (SQ) and recognition quality (RQ) into a single score: PQ = SQ * RQ. SQ measures the average IoU overlap for matched instances, while RQ measures classification accuracy for matched instances (TP / (TP + FP + FN)). Unmatched predictions or ground truths are penalized.
• Segmentation Quality (SQ) — range: [0, 1]
  • Measures the average IoU overlap between matched predicted and ground truth instances.
• Recognition Quality (RQ) — range: [0, 1]
  • Measures the classification accuracy of matched instances (true positives / (true positives + false positives + false negatives)).
• F1 / wF1 — range: [0, 1]
  • F1 score for semantic spotting; weighted F1 (wF1) accounts for class frequency.
• AP50 / AP75 / mAP — range: [0, 1]
  • Average Precision at IoU thresholds 0.50 and 0.75 for instance spotting; mAP is the mean across thresholds.

Input / output format

Input: Architectural CAD drawing chunks (images or point clouds), optionally accompanied by layer and color prior information.

Output: Predicted instance masks/boxes and semantic class labels, categorized into 'Thing' (discrete symbols) and 'Stuff' (continuous regions).

Scoring recipe

def compute_pq(preds, gts):
    matched = match_by_iou(preds, gts, thresh=0.5)
    sq = sum(iou(p, g) for p, g in matched) / len(matched) if matched else 0
    tp = len(matched)
    fp = len(preds) - tp
    fn = len(gts) - tp
    rq = tp / (tp + fp + fn) if (tp + fp + fn) > 0 else 0
    return sq * rq, sq, rq

Common pitfalls

• The evaluation explicitly separates results into 'Thing' (discrete symbols) and 'Stuff' (continuous architectural elements), requiring separate reporting and analysis.
• Performance heavily depends on CAD prior information (layer/color); the authors emphasize results without priors as more realistic, so comparing 'w/' vs 'w/o' columns is critical.
• Metrics are reported as percentages in tables (e.g., 90.1) but mathematically represent [0, 1] values.

Evidence (verbatim from paper)

Following the definition of panoptic symbol spotting, we evaluate the model performance using Panoptic Quality (PQ), Segmentation Quality (SQ), and Recognition Quality (RQ). The formulation of these metrics can be found in [[11]].

Citation

@misc{luo2025archcad400k,
  title={ArchCAD-400K: An Open Large-Scale Architectural CAD Dataset and New Baseline for Panoptic Symbol Spotting},
  author={Luo et al. (2025)},
  year={2025},
  note={arXiv:2503.22346}
}

• arXiv: 2503.22346