Poet Weather Calibration Eval

poet-weather-calibration-eval

Improving medium-range ensemble weather forecasts with hierarchical ensemble transformers — Ben Bouallègue et al. (2023) (arXiv:2303.17195, 2023)

What this evaluates

Evaluates the ability of a hierarchical transformer (PoET) to post-process and calibrate medium-range ensemble weather forecasts for 2m temperature and precipitation, compared to a baseline method (MBM) and raw ensemble outputs.

Datasets

• ECMWF ensemble forecasts — total ?; splits: train (-1), test (-1)

Metrics

• CRPS (primary) — range: other
  • Continuous Ranked Probability Score. For an ensemble forecast with M members $x_i$ and observation $y$, computed as $\frac{1}{M} \sum_{i=1}^M |x_i - y| - \frac{1}{2M^2} \sum_{i=1}^M \sum_{j=1}^M |x_i - x_j|$. Lower values indicate better probabilistic accuracy.
• CRPSS — range: [0, 1]
  • Continuous Ranked Probability Skill Score. Computed as $1 - \frac{CRPS_{model}}{CRPS_{reference}}$, where the reference is the raw ensemble forecast.
• ES — range: other
  • Energy Score. A multivariate generalization of CRPS applied over the time dimension, computed over 2 consecutive time steps for each pair of time steps separately.
• BS — range: [0, 1]
  • Brier Score. Assesses probability forecast performance for precipitation events exceeding 1mm and 10mm.
• BSS — range: [0, 1]
  • Brier Skill Score. Computed as $1 - \frac{BS_{model}}{BS_{reference}}$ using the raw ensemble as reference.

Input / output format

Input: Gridded atmospheric predictors (e.g., 2m temperature, 850hPa temperature, 500hPa geopotential, winds, cloud cover, orography, land-sea mask, insolation) for lead times 6h to 96h at 6-hour intervals.

Output: Calibrated post-processed forecast fields (2m temperature or precipitation) at each grid point.

Scoring recipe

def compute_crps(ensemble_preds, obs):
    mae = np.mean(np.abs(ensemble_preds - obs), axis=0)
    mae_inter = np.mean(np.abs(ensemble_preds[:, None, ...] - ensemble_preds[None, :, ...]), axis=(0, 1))
    return mae - 0.5 * mae_inter

def compute_skill(crps_model, crps_ref):
    return 1.0 - (crps_model / crps_ref)

# Aggregation
scores = compute_crps(predictions, observations)
# Weight by cosine of latitude when aggregating globally
weighted_scores = scores * np.cos(np.radians(latitudes))
final_score = np.mean(weighted_scores)

Common pitfalls

• Using Gaussian CRPS (gCRPS) for precipitation instead of kernel CRPS (kCRPS), which is required due to precipitation's non-Gaussian distribution.
• Forgetting to apply cosine-of-latitude weighting when aggregating global scores, which biases results toward high latitudes.
• Comparing PoET directly to MBM without accounting for MBM's time-of-year windowing, which makes MBM's training data forecast-specific.

Evidence (verbatim from paper)

The continuous ranked probability score (CRPS) is computed to assess the ensemble as a probabilistic forecast. Forecast performance in a multi-dimensional space is assessed using the energy score (ES), a generalization of the CRPS to the multivariate case. ES is applied over the time dimension, computed over 2 consecutive time steps for each pair of time steps separately. Additionally, for precipitation, probability forecast performance for pre-defined events is assessed with the Brier score (BS). We consider 2 precipitation events: 6-hourly precipitation exceeding 1mm and 10mm. The relative skill of a forecast with respect to a reference forecast is estimated with the help of skill scores. In the following, we compute the continuous ranked probability skill score (CRPSS), the energy skill score (ESS), and the Brier skill score (BSS) using the raw ensemble forecast as a reference.

Citation

@misc{benbouallègue2023improving,
  title={Improving medium-range ensemble weather forecasts with hierarchical ensemble transformers},
  author={Ben Bouallègue et al. (2023)},
  year={2023},
  note={arXiv:2303.17195}
}

• arXiv: 2303.17195