# Stormcast Forecast Eval

> Evaluates the ability of a generative diffusion model to emulate kilometer-scale atmospheric convection and precipitation forecasting. It probes spatial-temporal forecast skill, multi-variable physical consistency (e.g., updrafts, cold pools), and spectral fidelity over 1-6 hour lead times. Use when the user wants to benchmark on ERA5, HRRR, MRMS, or asks about evaluating this task. Reports FSS.

- Skill: `qhjqhj00/stormcast-forecast-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/stormcast-forecast-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/stormcast-forecast-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/stormcast-forecast-eval

---


# stormcast-forecast-eval

> Kilometer-Scale Convection Allowing Model Emulation using Generative Diffusion Modeling — Pathak et al. (2024) (arXiv:2408.10958, 2024)

## What this evaluates

Evaluates the ability of a generative diffusion model to emulate kilometer-scale atmospheric convection and precipitation forecasting. It probes spatial-temporal forecast skill, multi-variable physical consistency (e.g., updrafts, cold pools), and spectral fidelity over 1-6 hour lead times.

## Datasets

- **ERA5** — total ?; splits: (unstated)
- **HRRR** — total ?; splits: (unstated)
- **MRMS** — total ?; splits: (unstated)

## Metrics

- `FSS` **(primary)** — range: [0, 1]
  - Fractions Skill Score. Computed by binarizing forecast and reference fields at specified thresholds (20, 30, 40 dBZ), applying spatial pooling (15 or 45 km windows), and calculating fractional overlap: 1 - sum((F-R)^2) / sum(F^2 + R^2).
- `RMSE` — range: other
  - Root Mean Square Error computed relative to the HRRR analysis baseline across multiple atmospheric variables (radar reflectivity, wind, temperature, humidity).

## Input / output format

**Input**: 26 synoptic conditioning variables from ERA5 reanalysis, plus HRRR analysis fields at the initialization time step.

**Output**: 99 atmospheric state variables predicted at 1-hour intervals for 1-12 hour lead times, including composite radar reflectivity, horizontal wind components, temperature, and specific humidity at multiple vertical levels.

## Scoring recipe

```python
def compute_fss(forecast, reference, threshold, pool_size):
    f_bin = (forecast >= threshold).astype(float)
    r_bin = (reference >= threshold).astype(float)
    f_pool = spatial_average(f_bin, pool_size)
    r_pool = spatial_average(r_bin, pool_size)
    num = sum((f_pool - r_pool)**2)
    den = sum(f_pool**2 + r_pool**2)
    return 1 - (num / den)

def compute_rmse(forecast, reference):
    return sqrt(mean((forecast - reference)**2))
```

## Common pitfalls

- FSS values are highly sensitive to the chosen pooling window size and dBZ threshold; small changes drastically alter perceived skill.
- The 1-hour time resolution prevents direct temporal evolution analysis, requiring reliance on spatial/multivariate co-location checks instead.
- Ensemble Probability Matched Mean (PMM) masks individual member errors; evaluating single members vs. PMM yields different conclusions about skill.

## Evidence (verbatim from paper)

> The FSS of composite reflectivity forecasts are evaluated at thresholds of 20dBZ (light rain), 30dBZ (light-to-moderate rain) and 40dBZ (moderate rain). We compare the single member forecasts from StormCast (dotted lines) as well as the PMM of a five-member ensemble from the StormCast model (dashed lines) against the HRRR single member forecast baseline (solid lines).

## Citation

```bibtex
@misc{pathak2024stormcast,
  title={Kilometer-Scale Convection Allowing Model Emulation using Generative Diffusion Modeling},
  author={Pathak et al. (2024)},
  year={2024},
  note={arXiv:2408.10958}
}
```

- arXiv: 2408.10958