Exebench Eval

exebench-eval

ExEBench: Benchmarking Foundation Models on Extreme Earth Events — Zhao et al. (2025) (arXiv:2505.08529, 2025)

What this evaluates

Evaluates foundation models on detecting, forecasting, and monitoring extreme Earth events across seven categories including heatwaves, storms, floods, and wildfires. It probes model generalizability and transferability under data scarcity, distribution shift, and severe class imbalance across heterogeneous geospatial and meteorological modalities.

Datasets

• ExEBench — total ?; splits: train (-1), test (-1); repo https://github.com/zhaoshan2/EarthExtreme-Bench

Metrics

• Accuracy (ACC) (primary) — range: [0, 1]
  • Proportion of correctly classified pixels or forecasted time steps relative to the total number of samples.
• RMSE — range: [0, ∞)
  • Root Mean Squared Error between predicted and observed continuous values (e.g., temperature, precipitation).
• nRMSE — range: [0, 1]
  • Normalized RMSE, typically RMSE divided by the mean or range of observed values to enable cross-dataset comparison.
• HQE — range: [0, 1]
  • Hinge Quality Error, a metric used for evaluating probabilistic or continuous forecast calibration.
• POD, FAR, CSI, HSS — range: [0, 1]
  • Probability of Detection (TP/(TP+FN)), False Alarm Ratio (FP/(TP+FP)), Critical Success Index (TP/(TP+FP+FN)), and Heidke Skill Score for binary event detection.
• F1-score, IoU — range: [0, 1]
  • F1-score is the harmonic mean of precision and recall. IoU (Intersection over Union) measures overlap between predicted and ground truth segmentation masks.
• mF1, mIoU — range: [0, 1]
  • Mean F1-score and mean IoU averaged across multiple classes (e.g., non-flooded, open-flooded, urban flooded).

Input / output format

Input: Multimodal geospatial sequences including weather reanalysis variables (ERA5, TRMM, IMERG), radar precipitation rates, and SAR/multispectral imagery. Inputs vary in spatial resolution (20 m to 0.25°) and temporal steps, often normalized to fixed batch sizes (e.g., W×W or 512×512). Optional meta-information includes coordinates, resolution, and spectral bands.

Output: Task-specific predictions: single-step or multi-step continuous forecasts (temperature, precipitation, atmospheric variables), nowcasting trajectories, or pixel-wise segmentation masks (binary for burned scars, ternary for flood zones).

Scoring recipe

def compute_metrics(pred, gold, task_type):
    if task_type in ['forecasting', 'nowcasting']:
        acc = np.mean(pred == gold)
        rmse = np.sqrt(np.mean((pred - gold) ** 2))
        nrmse = rmse / np.mean(gold)
        return {'ACC': acc, 'RMSE': rmse, 'nRMSE': nrmse}
    elif task_type == 'segmentation':
        tp = np.sum((pred == 1) & (gold == 1))
        fp = np.sum((pred == 1) & (gold == 0))
        fn = np.sum((pred == 0) & (gold == 1))
        f1 = 2 * tp / (2 * tp + fp + fn)
        iou = tp / (tp + fp + fn)
        return {'F1': f1, 'IoU': iou}
    elif task_type == 'event_detection':
        pod = tp / (tp + fn)
        far = fp / (tp + fp)
        csi = tp / (tp + fp + fn)
        return {'POD': pod, 'FAR': far, 'CSI': csi}

Common pitfalls

• Severe class imbalance where non-target pixels (e.g., non-burned, non-flooded) vastly outnumber target pixels, biasing models toward majority classes.
• Distribution shift due to extreme events being rare in historical pre-training data, causing poor generalization to tail distributions.
• Heterogeneous spatial/temporal resolutions and modalities across tasks require careful normalization and alignment before unified evaluation.

Evidence (verbatim from paper)

Events from 2023 are designated as the test set, while all remaining events constitute the training set. ... Evaluation | ACC, RMSE, nRMSE, HQE | ACC, RMSE | ACC, RMSE | POD, FAR, CSI, HSS | | F-1, IoU | mF-1, mIoU

Citation

@misc{zhao2025exebench,
  title={ExEBench: Benchmarking Foundation Models on Extreme Earth Events},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2505.08529}
}

• arXiv: 2505.08529