Flairhub Eval

flairhub-eval

FLAIR-HUB: Large-scale Multimodal Dataset for Land Cover and Crop Mapping — Garioud et al. (2025) (arXiv:2506.07080, 2025)

What this evaluates

Evaluates semantic segmentation models for fine-grained land cover classification and crop type mapping using multi-sensor remote sensing imagery. It probes the model's ability to fuse spatial, spectral, and temporal modalities (aerial RGBI, SPOT, Sentinel-1/2, DEM) for pixel-level prediction at 20 cm resolution.

Datasets

• FLAIR-HUB — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection over Union computed per class and averaged. For land-cover, calculated over 15 valid classes (excluding ill-defined ones). For crop-type, calculated over classes present in the test set (excluding rice and other oilseed crops).
• O.A. — range: [0, 1]
  • Overall Accuracy, defined as the ratio of correctly predicted pixels to the total number of pixels across all classes.

Input / output format

Input: Multi-channel remote sensing imagery (AERIAL_RGBI or SPOT_RGBI with Infrared, Red, Green channels; DEM_ELEV; AERIAL_RLT-PAN) normalized using dataset statistics, aligned spatially and temporally for pixel-level semantic segmentation.

Output: Per-pixel class predictions for land cover (15 classes) or crop type classification.

Scoring recipe

def compute_metrics(predictions, ground_truth, valid_classes):
    ious = []
    correct = 0
    total = 0
    for cls in valid_classes:
        pred_cls = (predictions == cls)
        gt_cls = (ground_truth == cls)
        intersection = np.sum(pred_cls & gt_cls)
        union = np.sum(pred_cls | gt_cls)
        ious.append(intersection / union if union > 0 else 1.0)
        correct += np.sum(pred_cls & gt_cls)
        total += np.sum(gt_cls)
    mIoU = np.mean(ious)
    OA = correct / total
    return mIoU, OA

Common pitfalls

• mIoU is computed only over a subset of classes (15 for land cover, excluding ill-defined ones; crop-type excludes rice and other oilseed crops), not all available classes.
• Input channels for aerial/SPOT imagery are restricted to Infrared, Red, and Green (3 channels), not the full 4-channel RGBI.
• Normalization statistics are computed over the combined TRAIN and VAL partitions, not the full dataset or test set alone.

Evidence (verbatim from paper)

Metric: The performance of the semantic segmentation models is evaluated using the mean Intersection over Union (mIoU) and Overall Accuracy (O.A.) metrics. For the land-cover task, we exclude the ill-defined classes (see Table[VI]) and thus evaluate the results over the remaining 15 classes. For the LPIS crop-type task, two classes, rice and other oilseed crops, are absent from the test set, so mIoU is computed over the remaining classes.

Citation

@misc{garioud2025flairhub,
  title={FLAIR-HUB: Large-scale Multimodal Dataset for Land Cover and Crop Mapping},
  author={Garioud et al. (2025)},
  year={2025},
  note={arXiv:2506.07080}
}

• arXiv: 2506.07080