Maestro Eval

maestro-eval

MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Data — Labatie et al. (2025) (arXiv:2508.10894, 2025)

What this evaluates

This evaluation protocol assesses the transfer learning capability of self-supervised and supervised vision models on multimodal, multitemporal, and multispectral Earth observation data. It probes downstream performance on tree species classification and agricultural/land cover segmentation tasks across varying dataset scales and fusion strategies.

Datasets

• TreeSatAI-TS — total 50381; splits: train (-1), val (-1), test (-1)
• PASTIS-HD — total 433; splits: fold I (-1)
• FLAIR#2 — total 77762; splits: split 1 (-1)
• FLAIR-HUB — total 241100; splits: split 1 (-1)

Metrics

• weighted F1 score (primary) — range: [0, 100] percent
  • F1 score computed per class and averaged, weighted by the number of true instances per class. Calculated as 2 * (precision * recall) / (precision + recall + epsilon).
• mIoU (primary) — range: [0, 100] percent
  • Mean Intersection over Union across all semantic classes. Computed as the average of TP / (TP + FP + FN) for each class, where TP, FP, and FN are true positives, false positives, and false negatives.

Input / output format

Input: Multimodal image tiles containing aerial imagery (RGB+NIR), Sentinel-1/2 time series, and elevation data (DSM/DEM), processed into fixed-size patches for token-based model input.

Output: Per-pixel or per-tile class predictions: 15 multi-label tree species classes for TreeSatAI-TS, or 12/15 semantic segmentation masks for PASTIS-HD, FLAIR#2, and FLAIR-HUB.

Scoring recipe

def compute_miou(preds, gold, num_classes):
    ious = []
    for c in range(num_classes):
        tp = np.sum((preds == c) & (gold == c))
        fp = np.sum((preds == c) & (gold != c))
        fn = np.sum((preds != c) & (gold == c))
        iou = tp / (tp + fp + fn + 1e-6)
        ious.append(iou)
    return np.mean(ious) * 100

def compute_weighted_f1(preds, gold, num_classes):
    f1s, weights = [], []
    for c in range(num_classes):
        tp = np.sum((preds == c) & (gold == c))
        fp = np.sum((preds == c) & (gold != c))
        fn = np.sum((preds != c) & (gold == c))
        prec = tp / (tp + fp + 1e-6)
        rec = tp / (tp + fn + 1e-6)
        f1 = 2 * prec * rec / (prec + rec + 1e-6)
        f1s.append(f1)
        weights.append(np.sum(gold == c))
    return np.average(f1s, weights=weights) * 100

Common pitfalls

• Pre-training must strictly exclude test data; using the union of training and validation sets for self-supervised learning is required to prevent data leakage.
• Models must be evaluated on the exact dataset splits/folds specified (e.g., PASTIS-HD fold I, FLAIR split 1) to ensure comparability with reported baselines.
• Baseline foundation models require extensive hyperparameter tuning for fair comparison, as default settings may not reflect operational performance.

Evidence (verbatim from paper)

We report the weighted F1 score (%) on TreeSatAI-TS and the mIoU (%) on PASTIS-HD, FLAIR#2, and FLAIR-HUB.

Citation

@misc{labatie2025maestro,
  title={MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Data},
  author={Labatie et al. (2025)},
  year={2025},
  note={arXiv:2508.10894}
}

• arXiv: 2508.10894