Integrated Brier Score

integrated-brier-score

Combining predictive distributions for time-to-event outcomes in meteorology — Cunen et al. (2025) (arXiv:2503.19534, 2025)

What this evaluates

Evaluates the calibration and predictive accuracy of ensemble forecasting methods for time-to-event outcomes in meteorology. It compares how well different combination techniques predict the timing of events like the first hard freeze.

Datasets

• Meteorological time-to-event scenarios — total ?; splits: test (-1)

Metrics

• Mean PIT — range: [0, 1]
  • Probability Integral Transform values averaged across scenarios. For well-calibrated forecasts, PIT values should follow a uniform distribution on [0,1], so the mean should be close to 0.5.
• Standard deviation of PIT — range: [0, 1]
  • Standard deviation of PIT values across scenarios. Values closer to the theoretical uniform standard deviation (~0.289) indicate better calibration.
• Mean Integrated Brier Score (IBS) (primary) — range: [0, 1]
  • Integrated Brier Score averaged across scenarios. Measures the accuracy of predictive survival distributions; lower values indicate better predictive performance.

Input / output format

Input: Predictive survival distributions (or ensemble forecasts) for time-to-event outcomes, along with observed event times.

Output: Point estimates or full predictive distributions for the time until the first hard freeze (or similar time-to-event outcome).

Scoring recipe

For each scenario:
  pit_values = [F_i(t_i) for i in instances]  # F_i is predictive CDF, t_i is observed time
  mean_pit = mean(pit_values)
  std_pit = std(pit_values)
  ibs = integrated_brier_score(predictive_distributions, observed_times)
Return mean_pit, std_pit, and mean(ibs) across scenarios.

Common pitfalls

• PIT values near 0.5 indicate good calibration, but deviations can be subtle and require many scenarios to detect reliably.
• IBS is a proper scoring rule where lower is better; readers may mistakenly interpret higher scores as better performance.
• The evaluation aggregates results across 16 heterogeneous scenarios, which may mask method performance on specific meteorological conditions.

Evidence (verbatim from paper)

Table A3: Mean Integrated Brier scores (IBS) for 17 forecasting methods in 16 different scenarios. The lowest scores in each scenario are bold-faced.

Citation

@misc{cunen2025combining,
  title={Combining predictive distributions for time-to-event outcomes in meteorology},
  author={Cunen et al. (2025)},
  year={2025},
  note={arXiv:2503.19534}
}

• arXiv: 2503.19534