Stormcast Forecast Eval

stormcast-forecast-eval

Kilometer-Scale Convection Allowing Model Emulation using Generative Diffusion Modeling — Pathak et al. (2024) (arXiv:2408.10958, 2024)

What this evaluates

Evaluates the ability of a generative diffusion model to emulate kilometer-scale atmospheric convection and precipitation forecasting. It probes spatial-temporal forecast skill, multi-variable physical consistency (e.g., updrafts, cold pools), and spectral fidelity over 1-6 hour lead times.

Datasets

• ERA5 — total ?; splits: (unstated)
• HRRR — total ?; splits: (unstated)
• MRMS — total ?; splits: (unstated)

Metrics

• FSS (primary) — range: [0, 1]
  • Fractions Skill Score. Computed by binarizing forecast and reference fields at specified thresholds (20, 30, 40 dBZ), applying spatial pooling (15 or 45 km windows), and calculating fractional overlap: 1 - sum((F-R)^2) / sum(F^2 + R^2).
• RMSE — range: other
  • Root Mean Square Error computed relative to the HRRR analysis baseline across multiple atmospheric variables (radar reflectivity, wind, temperature, humidity).

Input / output format

Input: 26 synoptic conditioning variables from ERA5 reanalysis, plus HRRR analysis fields at the initialization time step.

Output: 99 atmospheric state variables predicted at 1-hour intervals for 1-12 hour lead times, including composite radar reflectivity, horizontal wind components, temperature, and specific humidity at multiple vertical levels.

Scoring recipe

def compute_fss(forecast, reference, threshold, pool_size):
    f_bin = (forecast >= threshold).astype(float)
    r_bin = (reference >= threshold).astype(float)
    f_pool = spatial_average(f_bin, pool_size)
    r_pool = spatial_average(r_bin, pool_size)
    num = sum((f_pool - r_pool)**2)
    den = sum(f_pool**2 + r_pool**2)
    return 1 - (num / den)

def compute_rmse(forecast, reference):
    return sqrt(mean((forecast - reference)**2))

Common pitfalls

• FSS values are highly sensitive to the chosen pooling window size and dBZ threshold; small changes drastically alter perceived skill.
• The 1-hour time resolution prevents direct temporal evolution analysis, requiring reliance on spatial/multivariate co-location checks instead.
• Ensemble Probability Matched Mean (PMM) masks individual member errors; evaluating single members vs. PMM yields different conclusions about skill.

Evidence (verbatim from paper)

The FSS of composite reflectivity forecasts are evaluated at thresholds of 20dBZ (light rain), 30dBZ (light-to-moderate rain) and 40dBZ (moderate rain). We compare the single member forecasts from StormCast (dotted lines) as well as the PMM of a five-member ensemble from the StormCast model (dashed lines) against the HRRR single member forecast baseline (solid lines).

Citation

@misc{pathak2024stormcast,
  title={Kilometer-Scale Convection Allowing Model Emulation using Generative Diffusion Modeling},
  author={Pathak et al. (2024)},
  year={2024},
  note={arXiv:2408.10958}
}

• arXiv: 2408.10958