qhjqhj00 treesatai-ts-eval
OmniSat: Self-Supervised Modality Fusion for Earth Observation — Astruc et al. (2024) (arXiv:2404.08351, 2024)
What this evaluates
Evaluates a model's ability to perform fine-grained tree species identification using multimodal Earth observation data, specifically leveraging temporal dynamics from optical and radar time series alongside high-resolution imagery.
Datasets
- TreeSatAI-TS — total 50381; splits: train (-1), test (-1); repo https://github.com/gastruc/OmniSat
Metrics
weighted F1(primary) — range: percent- Weighted average of per-class F1 scores, where each class's contribution is proportional to its support (number of true instances).
Input / output format
Input: Multimodal satellite tiles containing VHR RGB/NIR images, Sentinel-1 SAR time series, and Sentinel-2 optical time series, georeferenced to 60x60m areas.
Output: Multi-label classification vector indicating presence/absence across 20 tree species classes.
Scoring recipe
def compute_weighted_f1(gold, pred):
from sklearn.metrics import f1_score
return f1_score(gold, pred, average='weighted', zero_division=0) * 100
Common pitfalls
- Confusing single-date baselines with time-series inputs, which drastically underperform on temporal modalities.
- Using late feature fusion instead of end-to-end multimodal fusion, which fails to capture cross-modal interactions.
Evidence (verbatim from paper)
We report the weighted F1 for multi-label tree species classification on TreeSatAI (TSAI) and our extended TreeSatAI-TS (TSAI-TS) dataset when fine-tuning with 10% and 100% of training labels.
Citation
@misc{astruc2024omnisat,
title={OmniSat: Self-Supervised Modality Fusion for Earth Observation},
author={Astruc et al. (2024)},
year={2024},
note={arXiv:2404.08351}
}
- arXiv: 2404.08351