Climate Ood Robustness Eval

climate-ood-robustness-eval

Assessing the Robustness of Climate Foundation Models under No-Analog Distribution Shifts — Agana Navarro et al. (2026) (arXiv:2603.23043, 2026)

What this evaluates

Evaluates the out-of-distribution robustness of climate emulators under temporal extrapolation and cross-scenario forcing shifts. It probes whether models trained on historical climate data can accurately generalize to novel future regimes and extreme emission pathways without seeing them during training.

Datasets

• ClimateSet / CMIP6 GCM outputs — total ?; splits: train (-1), test_temporal (-1), test_SSP1-2.6 (-1), test_SSP5-8.5 (-1)

Metrics

• LL-RMSE (primary) — range: [0, ∞)
  • Root Mean Squared Error computed on log-transformed or link-function-transformed target variables (surface air temperature and precipitation). Lower values indicate better accuracy.

Input / output format

Input: Spatial-temporal climate data including forcing agents and atmospheric responses for surface air temperature (TAS) and precipitation (PR) from GCMs (EC-Earth3, MPI-ESM1-2-HR).

Output: Predicted values for TAS and PR.

Scoring recipe

def compute_ll_rmse(y_true, y_pred):
    return np.sqrt(np.mean((y_pred - y_true) ** 2))

def compute_percent_change(ll_rmse_id, ll_rmse_oob):
    return ((ll_rmse_oob - ll_rmse_id) / ll_rmse_id) * 100

Common pitfalls

• High absolute LL-RMSE values (0.8–1.1) are noted as a limitation due to standardized training without per-scenario hyperparameter tuning, which may mask intrinsic architectural fragility but ensures fair comparison.
• Potential data contamination from ClimaX's pre-training on CMIP6 historical data blurs the boundary between in-distribution and out-of-distribution evaluation.
• Precipitation shows consistently higher degradation than temperature due to stochastic hydrological physics, meaning temperature-only evaluations may overestimate model robustness.

Evidence (verbatim from paper)

The absolute LL-RMSE values achieved (ranging from 0.8 to 1.1) represent significantly lower accuracy than state-of-the-art emulators, which typically achieve errors in the 0.2–0.3 range. In contrast, Precipitation (PR) saw a consistent performance degradation across all architectures, with LL-RMSE increasing by up to 5.18%.

Citation

@misc{agana_navarro2026climateood,
  title={Assessing the Robustness of Climate Foundation Models under No-Analog Distribution Shifts},
  author={Agana Navarro et al. (2026)},
  year={2026},
  note={arXiv:2603.23043}
}

• arXiv: 2603.23043