Skyscenes Aerial Seg Eval

skyscenes-aerial-seg-eval

SkyScenes: A Synthetic Dataset for Aerial Scene Understanding — Khose et al. (2023) (arXiv:2312.06719, 2023)

What this evaluates

Evaluates semantic segmentation models trained on synthetic aerial imagery for their ability to generalize to real-world UAV datasets and adapt to varying environmental conditions like weather, time of day, and camera viewpoint.

Datasets

• SKYSCENES — total ?; splits: train (-1)
• UAVid — total ?; splits: test (-1)
• AEROSCAPES — total ?; splits: test (-1)
• ICG DRONE — total ?; splits: test (-1)
• SYNDrone — total ?; splits: train (-1)

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection over Union, computed as the average of IoU across all semantic classes. IoU per class is calculated as TP / (TP + FP + FN).
• Test IoU — range: [0, 1]
  • Per-class Intersection over Union reported for specific target classes (e.g., vehicle, human) to evaluate tail-class generalization.

Input / output format

Input: RGB aerial images (and optionally depth maps in multi-modal experiments) captured from UAVs at varying altitudes, pitch angles, weather conditions, and times of day.

Output: Per-pixel semantic segmentation masks assigning each pixel to a predefined class (e.g., road, building, vehicle, human, tree, low-veg, clutter).

Scoring recipe

def compute_miou(pred_masks, gt_masks, num_classes):
    ious = []
    for c in range(num_classes):
        pred_c = (pred_masks == c)
        gt_c = (gt_masks == c)
        intersection = np.logical_and(pred_c, gt_c).sum()
        union = np.logical_or(pred_c, gt_c).sum()
        iou = intersection / union if union > 0 else 0.0
        ious.append(iou)
    return np.mean(ious)

Common pitfalls

• Class vocabularies differ between synthetic and real datasets; models require explicit class merging/alignment before evaluation.
• Real-world datasets are captured at different heights and pitch angles; evaluation subsets must be aligned with corresponding synthetic viewpoint conditions.
• Low-shot experiments use specific fixed percentages (5%, 10%, 25%, 50%, 100%) of the real training set, not random splits.

Evidence (verbatim from paper)

We assess performance on 3 real-world aerial datasets - UAVid [27], AEROSCAPEs [34], ICG DRONE [20]. Since different datasets have different class vocabularies and definitions, for our experiments, we adapt the class vocabulary of the synthetic source dataset to that of the target real-world datasets (see Sec. C.1 in appendix for class merging and assignment schemes). Table 4: SKYSCENES augmented real data improves performance in low shot regimes. We compare DeepLabv2 [4], DAFormer [16], and Rein [54] models trained using varying percentages of labeled UAVid [27] images. Models are either trained jointly on SKYSCENES and UAVid (JT) or pretrained on SKYSCENES and finetuned on UAVid (FT). Augmenting real data with SKYSCENES enhances real-world generalization in low-shot scenarios. Real World mIoU (↑)

Citation

@misc{khose2023skyscenes,
  title={SkyScenes: A Synthetic Dataset for Aerial Scene Understanding},
  author={Khose et al. (2023)},
  year={2023},
  note={arXiv:2312.06719}
}

• arXiv: 2312.06719