P3 Building Vectorization Eval

p3-building-vectorization-eval

The P$^3$ dataset: Pixels, Points and Polygons for Multimodal Building Vectorization — Sulzer et al. (2025) (arXiv:2505.15379, 2025)

What this evaluates

Evaluates multimodal building vectorization by predicting building outlines from fused aerial imagery and LiDAR point clouds. Probes geometric accuracy, boundary precision, polygon complexity, and computational efficiency across diverse urban environments.

Datasets

• P$^3$ dataset — total ?; splits: test (-1), switzerland_subset (-1); repo https://github.com/raphaelsulzer/PixelsPointsPolygons

Metrics

• IoU (primary) — range: [0, 1]
  • Intersection over Union between predicted and ground truth polygon areas. Measures combined accuracy and completeness.
• AP — range: [0, 1]
  • Average Precision computed over multiple IoU thresholds. Measures the accuracy of predicted polygon sets.
• AR — range: [0, 1]
  • Average Recall computed over multiple IoU thresholds. Measures the completeness of predicted polygon sets.
• POLIS — range: other
  • Symmetric distance between each predicted polygon vertex and its closest point on the ground truth boundary, and vice versa.
• HD — range: other
  • Hausdorff distance measuring the maximum boundary distance between predicted and ground truth polygons.
• CD — range: other
  • Chamfer distance measuring the average boundary distance between predicted and ground truth polygons.
• MTA — range: other
  • Maximum tangent angle measuring the maximum angular error between predicted and ground truth polygon edges.
• DoF — range: other
  • Normalized degree of freedom measuring the regularity and symmetry of polygon edges. Lower scores indicate simpler, more regular polygons.

Input / output format

Input: Paired aerial imagery (224×224 pixels, 25 cm GSD) and LiDAR point clouds, voxelized into 784 pillars aligned with image patches.

Output: Sequence of vertex coordinates forming predicted building polygons, optionally with permutation/grouping matrices for end-to-end models.

Scoring recipe

def evaluate(preds, gts):
    tile_metrics = []
    for pred_tile, gt_tile in zip(preds, gts):
        m = {}
        m['IoU'] = mean_iou(pred_tile, gt_tile)
        m['AP'], m['AR'] = compute_ap_ar(pred_tile, gt_tile, iou_threshs=[0.5, 0.75])
        valid = [(p, g) for p, g in zip(pred_tile, gt_tile) if iou(p, g) >= 0.5]
        m['POLIS'] = mean(symmetric_vertex_dist(p, g) for p, g in valid)
        m['HD'] = mean(max_boundary_dist(p, g) for p, g in valid)
        m['CD'] = mean(mean_boundary_dist(p, g) for p, g in valid)
        m['MTA'] = mean(max_edge_angle_error(p, g) for p, g in valid)
        m['NR'] = mean(abs(len(p)-len(g))/len(g) for p, g in zip(pred_tile, gt_tile))
        m['C-IoU'] = m['NR'] * m['IoU']
        m['DoF'] = compute_normalized_dof(pred_tile)
        tile_metrics.append(m)
    return {k: mean([t[k] for t in tile_metrics]) for k in tile_metrics[0]}

Common pitfalls

• Boundary metrics (POLIS, HD, CD, MTA) are strictly filtered to polygon pairs with IoU ≥ 0.5, which can hide poor predictions that fail the threshold.
• Metrics are averaged per tile before global averaging, potentially biasing results if tile sizes or building densities vary across regions.
• End-to-end models require correct vertex ordering and grouping; incorrect permutation matrices lead to artificially low IoU and boundary distances.

Evidence (verbatim from paper)

POLIS, HD, CD and MTA are computed for ground truth and predicted polygon pairs with a minimum IoU (intersection over union) of 0.5. They are important measures for practitioners because they correlate with a high visual similarly of predicted and ground truth polygons.

Citation

@misc{sulzer2025p3dataset,
  title={The P$^3$ dataset: Pixels, Points and Polygons for Multimodal Building Vectorization},
  author={Sulzer et al. (2025)},
  year={2025},
  note={arXiv:2505.15379}
}

• arXiv: 2505.15379