maestro-eval
MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Data — Labatie et al. (2025) (arXiv:2508.10894, 2025)
What this evaluates
This evaluation protocol assesses the transfer learning capability of self-supervised and supervised vision models on multimodal, multitemporal, and multispectral Earth observation data. It probes downstream performance on tree species classification and agricultural/land cover segmentation tasks across varying dataset scales and fusion strategies.
Datasets
- TreeSatAI-TS — total 50381; splits: train (-1), val (-1), test (-1)
- PASTIS-HD — total 433; splits: fold I (-1)
- FLAIR#2 — total 77762; splits: split 1 (-1)
- FLAIR-HUB — total 241100; splits: split 1 (-1)
Metrics
weighted F1 score (primary) — range: [0, 100] percent
- F1 score computed per class and averaged, weighted by the number of true instances per class. Calculated as 2 * (precision * recall) / (precision + recall + epsilon).
mIoU (primary) — range: [0, 100] percent
- Mean Intersection over Union across all semantic classes. Computed as the average of TP / (TP + FP + FN) for each class, where TP, FP, and FN are true positives, false positives, and false negatives.
Input / output format
Input: Multimodal image tiles containing aerial imagery (RGB+NIR), Sentinel-1/2 time series, and elevation data (DSM/DEM), processed into fixed-size patches for token-based model input.
Output: Per-pixel or per-tile class predictions: 15 multi-label tree species classes for TreeSatAI-TS, or 12/15 semantic segmentation masks for PASTIS-HD, FLAIR#2, and FLAIR-HUB.
Scoring recipe
def compute_miou(preds, gold, num_classes):
ious = []
for c in range(num_classes):
tp = np.sum((preds == c) & (gold == c))
fp = np.sum((preds == c) & (gold != c))
fn = np.sum((preds != c) & (gold == c))
iou = tp / (tp + fp + fn + 1e-6)
ious.append(iou)
return np.mean(ious) * 100
def compute_weighted_f1(preds, gold, num_classes):
f1s, weights = [], []
for c in range(num_classes):
tp = np.sum((preds == c) & (gold == c))
fp = np.sum((preds == c) & (gold != c))
fn = np.sum((preds != c) & (gold == c))
prec = tp / (tp + fp + 1e-6)
rec = tp / (tp + fn + 1e-6)
f1 = 2 * prec * rec / (prec + rec + 1e-6)
f1s.append(f1)
weights.append(np.sum(gold == c))
return np.average(f1s, weights=weights) * 100
Common pitfalls
- Pre-training must strictly exclude test data; using the union of training and validation sets for self-supervised learning is required to prevent data leakage.
- Models must be evaluated on the exact dataset splits/folds specified (e.g., PASTIS-HD fold I, FLAIR split 1) to ensure comparability with reported baselines.
- Baseline foundation models require extensive hyperparameter tuning for fair comparison, as default settings may not reflect operational performance.
Evidence (verbatim from paper)
We report the weighted F1 score (%) on TreeSatAI-TS and the mIoU (%) on PASTIS-HD, FLAIR#2, and FLAIR-HUB.
Citation
@misc{labatie2025maestro,
title={MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Data},
author={Labatie et al. (2025)},
year={2025},
note={arXiv:2508.10894}
}
1---2name: maestro-eval3description: This evaluation protocol assesses the transfer learning capability of self-supervised and supervised vision models on multimodal, multitemporal, and multispectral Earth observation data. It probes downstream performance on tree species classification and agricultural/land cover segmentation tasks across varying dataset scales and fusion strategies. Use when the user wants to benchmark on TreeSatAI-TS, PASTIS-HD, FLAIR#2, FLAIR-HUB, or asks about evaluating this task. Reports weighted F1 score, mIoU.4metadata:5 source: https://skillmd.com/skills/qhjqhj00/maestro-eval6 skill_kind: dataset_eval7 source_arxiv: 2508.108948 bibtex_key: labatie2025maestro9 confidence: high10---1112# maestro-eval1314> MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Data — Labatie et al. (2025) (arXiv:2508.10894, 2025)1516## What this evaluates1718This evaluation protocol assesses the transfer learning capability of self-supervised and supervised vision models on multimodal, multitemporal, and multispectral Earth observation data. It probes downstream performance on tree species classification and agricultural/land cover segmentation tasks across varying dataset scales and fusion strategies.1920## Datasets2122- **TreeSatAI-TS** — total 50381; splits: train (-1), val (-1), test (-1)23- **PASTIS-HD** — total 433; splits: fold I (-1)24- **FLAIR#2** — total 77762; splits: split 1 (-1)25- **FLAIR-HUB** — total 241100; splits: split 1 (-1)2627## Metrics2829- `weighted F1 score` **(primary)** — range: [0, 100] percent30 - F1 score computed per class and averaged, weighted by the number of true instances per class. Calculated as 2 * (precision * recall) / (precision + recall + epsilon).31- `mIoU` **(primary)** — range: [0, 100] percent32 - Mean Intersection over Union across all semantic classes. Computed as the average of TP / (TP + FP + FN) for each class, where TP, FP, and FN are true positives, false positives, and false negatives.3334## Input / output format3536**Input**: Multimodal image tiles containing aerial imagery (RGB+NIR), Sentinel-1/2 time series, and elevation data (DSM/DEM), processed into fixed-size patches for token-based model input.3738**Output**: Per-pixel or per-tile class predictions: 15 multi-label tree species classes for TreeSatAI-TS, or 12/15 semantic segmentation masks for PASTIS-HD, FLAIR#2, and FLAIR-HUB.3940## Scoring recipe4142```python43def compute_miou(preds, gold, num_classes):44 ious = []45 for c in range(num_classes):46 tp = np.sum((preds == c) & (gold == c))47 fp = np.sum((preds == c) & (gold != c))48 fn = np.sum((preds != c) & (gold == c))49 iou = tp / (tp + fp + fn + 1e-6)50 ious.append(iou)51 return np.mean(ious) * 1005253def compute_weighted_f1(preds, gold, num_classes):54 f1s, weights = [], []55 for c in range(num_classes):56 tp = np.sum((preds == c) & (gold == c))57 fp = np.sum((preds == c) & (gold != c))58 fn = np.sum((preds != c) & (gold == c))59 prec = tp / (tp + fp + 1e-6)60 rec = tp / (tp + fn + 1e-6)61 f1 = 2 * prec * rec / (prec + rec + 1e-6)62 f1s.append(f1)63 weights.append(np.sum(gold == c))64 return np.average(f1s, weights=weights) * 10065```6667## Common pitfalls6869- Pre-training must strictly exclude test data; using the union of training and validation sets for self-supervised learning is required to prevent data leakage.70- Models must be evaluated on the exact dataset splits/folds specified (e.g., PASTIS-HD fold I, FLAIR split 1) to ensure comparability with reported baselines.71- Baseline foundation models require extensive hyperparameter tuning for fair comparison, as default settings may not reflect operational performance.7273## Evidence (verbatim from paper)7475> We report the weighted F1 score (%) on TreeSatAI-TS and the mIoU (%) on PASTIS-HD, FLAIR#2, and FLAIR-HUB.7677## Citation7879```bibtex80@misc{labatie2025maestro,81 title={MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Data},82 author={Labatie et al. (2025)},83 year={2025},84 note={arXiv:2508.10894}85}86```8788- arXiv: 2508.10894