S2s AI Challenge Eval

s2s-ai-challenge-eval

Deep learning for post-processing global probabilistic forecasts on sub-seasonal time scales — Horat et al. (2023) (arXiv:2306.15956, 2023)

What this evaluates

Evaluates the skill of deep learning post-processing models for global sub-seasonal temperature and precipitation forecasts against climatological baselines and ECMWF recalibrated forecasts. It probes the ability of spatial CNN architectures to correct systematic errors and produce well-calibrated probabilistic tercile predictions over a 2–4 week horizon.

Datasets

• S2S AI Challenge test set (2020) — total 53; splits: test (53); repo https://github.com/HoratN/pp-s2s

Metrics

• RPSS (primary) — range: [-1, 1]
  • RPSS = 1 − (RPS_model / RPS_climatology), where RPS is the Ranked Probability Score averaged over the 53 weekly forecasts. Positively oriented; higher values indicate better skill relative to climatology.

Input / output format

Input: Global spatial forecast fields (ensemble mean quantities) from ECMWF for temperature and precipitation.

Output: Probabilistic tercile forecasts (probabilities for above, near, and below normal categories) for each grid cell.

Scoring recipe

def compute_rpss(predictions, observations, climatology_probs):
    # predictions, observations, climatology_probs: shape (num_forecasts, num_grid_cells, 3_terciles)
    rps_model = mean((predictions - observations) ** 2, axis=0)
    rps_clim = mean((climatology_probs - observations) ** 2, axis=0)
    rpss_grid = 1 - (rps_model / rps_clim)
    # Weighted average over land cells, excluding Antarctica and dry precip regions
    rpss_global = weighted_mean(rpss_grid, weights=land_area_fraction)
    return rpss_global

Common pitfalls

• Only land grid cells are considered; sea cells are excluded due to lack of observations.
• Antarctica is explicitly excluded from the aggregated RPSS calculation.
• For precipitation, very dry regions (lower tercile edge < 0.01 mm/m^2) are omitted from evaluation.
• The metric is averaged over exactly 53 weekly forecasts issued on Thursdays in 2020.

Evidence (verbatim from paper)

Following the setup of the S2S AI Challenge, we evaluate the proposed post-processing methods based on the ranked probability skill score (RPSS) for the test set (year 2020), using the climatological forecast as a reference. The RPSS is based on a strictly proper scoring rule and introduced in detail in Appendix A. It is positively oriented, i.e., larger values indicate better forecasts. Following the challenge configuration, only land grid cells are considered for the computation of the spatially aggregated RPSS scores since no observations are available for sea grid cells. Antarctica is not included in the aggregated RPSS. For precipitation, also very dry regions (lower tercile edge smaller than $0.01\text{,}\mathrm{m}\mathrm{m}\mathrm{/}\mathrm{m}^{2}$ are not taken into account.

Citation

@misc{horat2023deeplearning,
  title={Deep learning for post-processing global probabilistic forecasts on sub-seasonal time scales},
  author={Horat et al. (2023)},
  year={2023},
  note={arXiv:2306.15956}
}

• arXiv: 2306.15956