Climatebench M Eval

climatebench-m-eval

ClimateBench-M: A Multi-Modal Climate Data Benchmark with a Simple Generative Method — Fu et al. (2025) (arXiv:2504.07394, 2025)

What this evaluates

Evaluates AI models on multi-modal climate data across three tasks: tensor time-series forecasting, extreme weather anomaly detection, and crop classification from satellite imagery. It probes spatial-temporal alignment, generative data synthesis, and robustness to highly imbalanced rare events.

Datasets

• ClimateBench-M — total ?; splits: train (-1), test (-1); repo https://github.com/iDEA-iSAIL-Lab-UIUC/ClimateBench-M

Metrics

• Accuracy (Acc) — range: [0, 1]
  • Evaluates the overlap between prediction and ground-truth: Acc = a/b, where a is the number of correct predictions and b is the total number of samples.
• Mean Absolute Error (MAE) (primary) — range: [0, inf)
  • Assesses the difference between prediction and ground truth. Standard MAE calculation over forecasted tensor values.
• Intersection of Union (IoU) — range: [0, 1]
  • Measures the ratio of the intersection of two sets over their union: IoU = |A∩B| / |A∪B|, where A is the prediction set and B is the ground-truth set.
• Area Under the Receiver Operating Characteristic Curve (AUC-ROC) — range: [0, 1]
  • Quantifies the ability to distinguish between classes by measuring the area under the ROC curve: AUC-ROC = ∫ b(a) da, where a and b are TPR and FPR respectively.

Input / output format

Input: Forecasting: past 24-hour tensor time series. Anomaly Detection: hourly forecasted weather features at specific locations. Crop Classification: concatenated satellite images across time channels for a given location.

Output: Forecasting: future 24-hour tensor time series. Anomaly Detection: anomaly score/condition based on feature-wise generation probability. Crop Classification: predicted crop class or segmentation mask.

Scoring recipe

def compute_metrics(pred, gold):
    acc = np.sum(pred == gold) / len(gold)
    mae = np.mean(np.abs(pred - gold))
    intersection = len(set(pred) & set(gold))
    union = len(set(pred) | set(gold))
    iou = intersection / union if union > 0 else 0.0
    fpr, tpr, _ = roc_curve(gold, pred)
    auc = np.trapz(tpr, fpr)
    return {'Acc': acc, 'MAE': mae, 'IoU': iou, 'AUC-ROC': auc}

Common pitfalls

• Baselines for anomaly detection use ground-truth observations while SGM uses forecast features, creating an asymmetric evaluation setup.
• Tensor time-series baselines are flattened to multi-variate series, with results averaged across all 238 locations.
• Crop segmentation baselines lack native temporal modeling, requiring manual channel concatenation of multi-temporal images.
• Extreme weather anomalies are highly rare (~0.45% frequency), making standard supervised baselines prone to poor performance.

Evidence (verbatim from paper)

We measure the performance of the baseline methods as well as the proposed method on the ClimateBench-M with respect to the following metrics: (1) Accuracy (Acc): It evaluates the overlap between the prediction and the ground-truth, i.e., Acc = a/b, where a is the number of correct prediction and b is the total number of samples. (2) Mean Absolute Error (MAE): It assess the difference between the prediction and the ground truth... (3) Intersection of Union (IoU): It measures the ratio of the intersection of two sets over the union of two sets... (4) Area Under the Receiver Operating Characteristic Curve (AUC-ROC): It quantifies the ability of a model to distinguish between classes by measuring the area under the ROC curve.

Citation

@misc{fu2025climatebenchm,
  title={ClimateBench-M: A Multi-Modal Climate Data Benchmark with a Simple Generative Method},
  author={Fu et al. (2025)},
  year={2025},
  note={arXiv:2504.07394}
}

• arXiv: 2504.07394