Flair One Eval

flair-one-eval

FLAIR #1: semantic segmentation and domain adaptation dataset — Garioud et al. (2022) (arXiv:2211.12979, 2022)

What this evaluates

Evaluates the ability of models to perform high-resolution land-cover semantic segmentation on aerial imagery. It probes robustness to spatial, temporal, and multi-sensor domain shifts, as well as handling radiometric inconsistencies and phenological variations across diverse landscapes.

Datasets

• FLAIR-one — total 77412; splits: train (61712), test (15700); repo https://github.com/IGNF/odeon-landcover

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection over Union, computed as the average of the Intersection over Union (IoU) across all 13 semantic classes. IoU for a class is the ratio of correctly predicted pixels to the union of predicted and ground truth pixels.

Input / output format

Input: High-resolution (0.2m) aerial image patches, optionally accompanied by acquisition metadata.

Output: Per-pixel semantic segmentation mask assigning one of 13 land-cover classes.

Scoring recipe

def compute_miou(preds, targets, num_classes=13):
    ious = []
    for c in range(num_classes):
        tp = ((preds == c) & (targets == c)).sum()
        fp = ((preds == c) & (targets != c)).sum()
        fn = ((preds != c) & (targets == c)).sum()
        ious.append(tp / (tp + fp + fn + 1e-6))
    return sum(ious) / len(ious)

Common pitfalls

• Models struggle with low-frequency classes (e.g., bare soil, coniferous), achieving IoU < 0.4.
• High inter-class confusion occurs between semantically similar categories like herbaceous vegetation and agricultural land.
• Metadata integration and standard geometric augmentations did not yield performance gains in the reported baseline, suggesting careful modality fusion or class-aware augmentation is needed.

Evidence (verbatim from paper)

The results obtained using 61,712 patches for training, and testing on the remaining 15,700 patches of the FLAIR-one dataset are reported in Table II. The given results are average and their standard deviation of $5\mathrm{mIoU}$ scores obtained for 5 runs in a given configuration. Detailed per-class IoU results for the baseline (without metadata integration or data augmentations) are illustrated in Figure 9.

Citation

@misc{garioud2022flair,
  title={FLAIR #1: semantic segmentation and domain adaptation dataset},
  author={Garioud et al. (2022)},
  year={2022},
  note={arXiv:2211.12979}
}

• arXiv: 2211.12979