Neuralgcm Climate Eval

neuralgcm-climate-eval

Neural General Circulation Models for Weather and Climate — Kochkov et al. (2023) (arXiv:2311.07222, 2023)

What this evaluates

Evaluates the model's ability to simulate historical global temperature trends and spatial temperature biases over multi-decadal climate simulations, and tests its generalization to warmer climate scenarios. It probes long-term stability, physical consistency, and response to prescribed sea surface temperature forcing.

Datasets

• ERA5 — total ?; splits: test (-1)

Metrics

• RMSB (850hPa temperature) (primary) — range: other
  • Root Mean Square Bias calculated as the square root of the mean squared difference between predicted and observed 850hPa temperatures, averaged over the 1981-2014 period.

Input / output format

Input: Initial atmospheric state and prescribed sea surface temperatures (SST) for AMIP-like simulations.

Output: Time-series of 3D atmospheric temperature fields, specifically aggregated to 850hPa temperature and vertical tropical profiles.

Scoring recipe

def compute_rmsb(predictions, observations):
    # predictions, observations: arrays of 850hPa temperature over 1981-2014
    diff = predictions - observations
    rmsb = np.sqrt(np.mean(diff ** 2))
    return rmsb

Common pitfalls

• Analysis is restricted to only the 22 out of 37 runs that remained stable for the full 40-year period; unstable trajectories are excluded from the main evaluation.
• Baseline comparisons shift significantly when global temperature bias is removed from CMIP6 runs (AMIP vs AMIP*), affecting relative performance claims.
• Generalization to +4K SST scenarios exhibits climate drift and diverges from expectations, limiting robustness claims for extreme warming.

Evidence (verbatim from paper)

When comparing spatial biases averaged over 1981-2014 we find that all 22 NeuralGCM- 2.8° runs have smaller bias than CMIP6 AMIP runs... (c) The root mean square bias (RMSB) of the 850hPa temperature averaged between 1981-2014 for 22 NeuralGCM- 2.8° AMIP runs (labeled NGCM), 22 CMIP6 AMIP experiments (labeled AMIP) and debiased 22 CMIP6 AMIP experiments (labeled as AMIP*).

Citation

@misc{kochkov2023neuralgcm,
  title={Neural General Circulation Models for Weather and Climate},
  author={Kochkov et al. (2023)},
  year={2023},
  note={arXiv:2311.07222}
}

• arXiv: 2311.07222