Weatherbench Probability Eval

weatherbench-probability-eval

WeatherBench Probability: A benchmark dataset for probabilistic medium-range weather forecasting along with deep learning baseline models — Garg et al. (2022) (arXiv:2205.00865, 2022)

What this evaluates

Evaluates the accuracy and reliability of probabilistic medium-range weather forecasting models against operational ensemble baselines. It probes how well deep learning methods capture uncertainty, calibration, and sharpness for key atmospheric variables.

Datasets

• WeatherBench Probability — total ?; splits: test (-1)

Metrics

• RMSE of ensemble mean — range: other
  • Root mean squared error between the ensemble mean forecast and observations. Lower values indicate better point accuracy.
• Spread-skill ratio — range: other
  • Ratio of the ensemble spread (standard deviation of members) to the RMSE of the ensemble mean. A value near 1 indicates optimal dispersion; <1 means underdispersive, >1 means overdispersive.
• CRPS (primary) — range: other
  • Continuous Ranked Probability Score, which evaluates both calibration and sharpness of probabilistic forecasts against observations. Lower values indicate better probabilistic skill.

Input / output format

Input: Atmospheric state variables (e.g., 500hPa geopotential height Z500, 850hPa temperature T850, 2m temperature T2M, total precipitation TP) for medium-range forecasting.

Output: Probabilistic forecasts: ensemble members, parametric distribution parameters, or categorical class probabilities.

Scoring recipe

def compute_metrics(pred_ensemble, pred_mean, obs):
    rmse = np.sqrt(np.mean((pred_mean - obs) ** 2))
    spread = np.std(pred_ensemble, axis=0)
    spread_skill = spread / rmse
    # CRPS for ensemble forecasts
    n = len(pred_ensemble)
    crps = (1/n) * np.sum(np.abs(pred_ensemble - obs)) - (1/(2*n**2)) * np.sum(np.abs(pred_ensemble[:, None] - pred_ensemble))
    return rmse, spread_skill, crps

Common pitfalls

• RMSE is not a suitable metric for highly intermittent and skewed fields like precipitation.
• TIGGE baseline results are not post-processed, which likely underestimates their calibration and spread-skill ratio.
• MC dropout severely underestimates uncertainty (underdispersion) unless dropout rate is carefully tuned.

Evidence (verbatim from paper)

The ensemble mean RMSE and the CRPS is lowest for a dropout rate of 0.1. The spread-skill ratio shows that the dropout ensemble is severely underdispersive with the spread being less than half of what it should be.

Citation

@misc{garg2022weatherbenchprobability,
  title={WeatherBench Probability: A benchmark dataset for probabilistic medium-range weather forecasting along with deep learning baseline models},
  author={Garg et al. (2022)},
  year={2022},
  note={arXiv:2205.00865}
}

• arXiv: 2205.00865