Lagged Ensembling Weather Eval

lagged-ensembling-weather-eval

A Practical Probabilistic Benchmark for AI Weather Models — Brenowitz et al. (2024) (arXiv:2401.15305, 2024)

What this evaluates

Evaluates the probabilistic forecasting skill and ensemble calibration of AI weather models by comparing them against a parameter-free lagged ensemble baseline. It probes whether models trained with long-lead-time objectives suffer from under-dispersion and poor variance calibration despite strong deterministic accuracy.

Datasets

• Atmospheric reanalysis / IFS HRES — total ?; splits: test (-1)

Metrics

• CRPS (primary) — range: other
  • Continuous Ranked Probability Score measuring the difference between the predictive cumulative distribution function and the observed step function. Lower values indicate better probabilistic calibration and accuracy.
• dRMSE — range: other
  • Deterministic Root Mean Squared Error between the single best forecast and observations.
• eRMSE — range: other
  • Ensemble Root Mean Squared Error between the ensemble mean forecast and observations.
• SER — range: other
  • Spread-Error Ratio: √((R+1)/R) * (Spread / eRMSE), where R is ensemble size. Measures calibration of ensemble spread relative to error.

Input / output format

Input: Initial atmospheric state fields (e.g., geopotential height, temperature, wind) at 0.25° resolution, used to generate forecasts at specified lead times (typically 2–8 days).

Output: Forecasted atmospheric fields at target lead times. For probabilistic evaluation, multiple forecast members are generated by initializing the model at different historical times (lagged ensemble).

Scoring recipe

def compute_crps(ensemble_preds, obs):
    # ensemble_preds: shape (M, H, W)
    # obs: shape (H, W)
    # Uses the typical biased estimator noted in the paper
    sorted_preds = np.sort(ensemble_preds, axis=0)
    crps = 0.0
    for i, pred in enumerate(sorted_preds):
        crps += (2 * (i + 1) / M - 1) * (pred - obs)
    crps /= M
    return np.mean(crps)

def compute_ser(ensemble_preds, obs, e_rmse):
    spread = np.std(ensemble_preds, axis=0)
    R = ensemble_preds.shape[0]
    ser = np.sqrt((R + 1) / R) * (spread / e_rmse)
    return np.mean(ser)

Common pitfalls

• Models optimized with long-lead-time MSE loss naturally converge to the conditional mean, artificially reducing ensemble variance and yielding under-dispersed forecasts that look good on deterministic RMSE but fail probabilistic calibration.
• Evaluating at lead times shorter than 2 days introduces initialization bias because IFS uses analysis (past weather) while evaluation uses reanalysis (future-informed), inflating early errors.
• Comparing AI models against NWP baselines without controlling for ensemble generation methods (e.g., initial condition perturbations) confounds architectural differences with ensemble technique differences.

Evidence (verbatim from paper)

The deterministic RMSE decreases significantly for models trained with more autoregressive timesteps, even appearing to approach the threshold of IFS deterministic skill limits (panel a). However, the CRPS and ensemble mean RMSE improve much less than the deterministic RMSE. The reason is due to plummeting ensemble spread (see Figure 4a-d) where for a given eRMSE the SFNO checkpoints optimized with long-lead time fine-tuning result in monotonic drops in ensemble variance, deviating from the approximately 1:1 error-spread calibration of the LEF IFS ensemble.

Citation

@misc{brenowitz2024practicalprobabilistic,
  title={A Practical Probabilistic Benchmark for AI Weather Models},
  author={Brenowitz et al. (2024)},
  year={2024},
  note={arXiv:2401.15305}
}

• arXiv: 2401.15305