Wildscenes Eval

wildscenes-eval

WildScenes: A Benchmark for 2D and 3D Semantic Segmentation in Large-scale Natural Environments — Vidanapathirana et al. (2023) (arXiv:2312.15364, 2023)

What this evaluates

Probes the capability of models to perform semantic segmentation on unstructured, large-scale natural environments using both 2D images and 3D LiDAR point clouds. It evaluates robustness to semantic ambiguity, clutter, and temporal environmental shifts in outdoor traversals.

Datasets

• WildScenes — total ?; splits: train (6051), val (283), test (2133)

Metrics

• mIoU (primary) — range: [0, 1]
  • Standard Mean Intersection over Union computed over 15 classes for 2D segmentation and 12 classes for 3D segmentation. IoU for a class is the intersection of predicted and ground truth pixels/points divided by their union.

Input / output format

Input: 2D: RGB images (cropped to 512x512). 3D: LiDAR point clouds.

Output: Per-pixel (2D) or per-point (3D) semantic class labels from a predefined set of 15 (2D) or 12 (3D) classes.

Scoring recipe

def compute_miou(preds, targets, num_classes):
    ious = []
    for c in range(num_classes):
        pred_mask = (preds == c)
        gt_mask = (targets == c)
        intersection = np.sum(pred_mask & gt_mask)
        union = np.sum(pred_mask | gt_mask)
        ious.append(intersection / union if union > 0 else 0.0)
    return np.mean(ious)

Common pitfalls

• 3D evaluation excludes 'sky', 'water', and 'other-terrain' classes due to LiDAR limitations or insufficient training points, resulting in 12 classes instead of the full 15.
• Some traversals are allocated to buffer regions and excluded from the train/val/test splits, meaning the reported split sizes do not cover the entire dataset.
• Backbones are initialized with ImageNet-1k/22k weights, which may not reflect zero-shot performance on out-of-domain natural scenes.

Evidence (verbatim from paper)

For evaluating the performance of a semantic segmentation method with respect to the ground truth label annotations, we use the standard Mean Intersection over Union (mIoU) metric for a set of 15 classes (2D) and 12 classes (3D).

Citation

@misc{vidanapathirana2023wildscenes,
  title={WildScenes: A Benchmark for 2D and 3D Semantic Segmentation in Large-scale Natural Environments},
  author={Vidanapathirana et al. (2023)},
  year={2023},
  note={arXiv:2312.15364}
}

• arXiv: 2312.15364