Parkseg12k Eval

parkseg12k-eval

A Pipeline and NIR-Enhanced Dataset for Parking Lot Segmentation — Qiam et al. (2024) (arXiv:2412.13179, 2024)

What this evaluates

This benchmark evaluates the ability of deep learning models to perform binary semantic segmentation of parking lots from satellite imagery. It specifically probes the model's capacity to generalize across different geographic locations and to leverage near-infrared (NIR) spectral data for improved contrast against vegetation and built environments.

Datasets

• ParkSeg12k — total 12617; splits: train (-1), val (-1), test (400); repo https://github.com/UTEL-UIUC/ParkSeg12k

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection over Union computed across all classes (parking and background). IoU is defined as the area of overlap between predicted and ground truth masks divided by their union: IoU(A,B) = |A∩B| / |A∪B|. The final score is the average IoU across classes.
• Pixel-wise accuracy (PW) — range: [0, 1]
  • The percentage of pixels correctly predicted as either background or parking lot out of the total number of pixels.

Input / output format

Input: Satellite image patches (RGB or 4-channel RGB+NIR) with corresponding geographic bounding boxes.

Output: Binary pixel-wise segmentation mask indicating 'parking' vs 'not parking', which is converted to GeoJSON polygons for post-processing.

Scoring recipe

def compute_metrics(pred_mask, gt_mask):
    # pred_mask, gt_mask: binary arrays (0=background, 1=parking)
    iou_parking = np.sum(pred_mask & gt_mask) / np.sum(pred_mask | gt_mask)
    iou_bg = np.sum((~pred_mask) & (~gt_mask)) / np.sum((~pred_mask) | (~gt_mask))
    mIoU = (iou_parking + iou_bg) / 2.0
    PW = np.mean(pred_mask == gt_mask)
    return {'mIoU': mIoU, 'PW': PW}

Common pitfalls

• Evaluation metrics in the paper are reported on post-processed polygon masks (after hole removal, edge simplification, and building/road subtraction), not on raw model outputs.
• The test set consists of 400 images from cities completely unseen during training/validation, so standard random splits will leak geographic bias.
• Class imbalance is severe (~21% parking vs ~79% background), requiring weighted loss or careful thresholding during inference.

Evidence (verbatim from paper)

Model performance is evaluated using two commonly-used metrics: • Pixel-wise accuracy (PW): Pixel-wise accuracy measures the percentage of pixels that are correctly predicted as either background or parking lot. • mean Intersection over Union (mIoU): This metric evaluates the average IoU for all classes (including background), measuring the overlap between the predicted mask and the ground truth mask: IoU(A,B)=|A∩B|/|A∪B| where |A∩B| is the area of overlap (intersection) between the predicted and ground truth masks, and |A∪B| is the area of union between the predicted and ground truth masks. This metric is computed for all the classes and the average is called mIoU.

Citation

@misc{qiam2024parkseg,
  title={A Pipeline and NIR-Enhanced Dataset for Parking Lot Segmentation},
  author={Qiam et al. (2024)},
  year={2024},
  note={arXiv:2412.13179}
}

• arXiv: 2412.13179