Probabilistic Rf Weather Forecast Eval

probabilistic-rf-weather-forecast-eval

A new paradigm for medium-range severe weather forecasts: probabilistic random forest-based predictions — Hill et al. (2022) (arXiv:2208.02383, 2022)

What this evaluates

Evaluates the skill of a probabilistic Random Forest model in forecasting severe thunderstorms (tornadoes, large hail, damaging winds) 4–8 days in advance using ensemble meteorological data. It probes the model's calibration, discrimination, and spatial coverage compared to human-generated SPC outlooks.

Datasets

• SPC Severe Weather Reports & GEFSv12 Reforecast — total ?; splits: train (-1), test (-1)

Metrics

• Brier Skill Score (BSS) (primary) — range: [-∞, 1]
  • BSS = 1 - (BS_fcsnt / BS_ref), where BS is the mean squared error between binary events and probabilistic forecasts, and BS_ref is the Brier score of a spatially and temporally smoothed long-term climatology.
• AUROC — range: [0, 1]
  • Area under the receiver operating characteristic curve, measuring the model's ability to discriminate between severe and non-severe events across all probability thresholds.

Input / output format

Input: Gridded (0.5° CONUS) meteorological variables from GEFSv12 (e.g., CAPE, PWAT, shear, temperature, humidity, wind) assembled via spatial averaging or time-lagging over a 24-hour forecast window, plus static lat/lon/julian day features. Labels are 0 (no severe), 1 (non-significant severe), or 2 (significant severe) based on interpolated SPC storm reports.

Output: Continuous probability (0–100%) of severe weather occurrence within 40 km of each grid point over a 24-hour period. For comparison, probabilities are discretized to SPC bin midpoints (7.5%, 22.5%, 65%).

Scoring recipe

# Calculate Brier Score (BS) per grid point
bs = mean((forecast_prob - binary_event) ** 2)
# Reference climatology BS (smoothed long-term report frequency)
bs_ref = mean((climatology_prob - binary_event) ** 2)
# Brier Skill Score
bss = 1 - (bs / bs_ref)
# Optional: AUROC
auroc = roc_auc_score(binary_event, forecast_prob)

Common pitfalls

• Discretization of continuous RF probabilities to match SPC's limited contour bins (7.5%, 22.5%, 65%) can obscure fine-grained calibration differences.
• The reference climatology for BSS uses a Gaussian-smoothed (15-day temporal, 120-km spatial) long-term average, not a simple historical mean, which affects baseline skill comparisons.
• Significant severe forecasts (class 2) are explicitly excluded from formal evaluation in this work, limiting assessment of high-impact event prediction.

Evidence (verbatim from paper)

Traditional methods used to quantify probabilistic prediction skill – e.g., Brier Skill Score (BSS), area under the receiver operating characteristic curve (AUROC), and reliability diagrams – are employed to evaluate the CSU-MLP and SPC forecasts. The Brier Score (BS) is a measure of the mean squared error between binary events and probabilistic forecasts. The BS can be converted to a skill score, the BSS, by comparing the BS of a forecast to a reference climatology BS. BSS ranges from −∞ to 1, with scores closer to 0 meaning the forecast skill is indistinguishable from the reference climatology (BS_ref): BSS = 1 - (BS_fcsnt / BS_ref).

Citation

@misc{hill2022probabilistic,
  title={A new paradigm for medium-range severe weather forecasts: probabilistic random forest-based predictions},
  author={Hill et al. (2022)},
  year={2022},
  note={arXiv:2208.02383}
}

• arXiv: 2208.02383