Probabilistic TS Forecasting Eval

probabilistic-ts-forecasting-eval

Learning to forecast: The probabilistic time series forecasting challenge — Bracher et al. (2022) (arXiv:2211.16171, 2022)

What this evaluates

Evaluates real-time probabilistic forecasting of financial and weather time series, probing a model's ability to quantify uncertainty via quantile modeling and maintain calibration over sequential submission rounds.

Datasets

• DAX, Wind, Temperature — total 199; splits: test (199); repo https://github.com/FK83/ptsfc_replication

Metrics

• skill score (primary) — range: other
  • Computed relative to a benchmark forecast using a quantile scoring function. A positive skill score indicates that the forecaster outperforms the benchmark.
• coverage rate — range: percent
  • The share of test-sample observations that fall within a specified prediction interval (e.g., 50% or 95% level).

Input / output format

Input: Historical time series data and optional external feature variables for targets (DAX, wind speed, temperature).

Output: Probabilistic forecasts in the form of prediction intervals or quantiles at multiple forecast horizons (e.g., 1-7 days for DAX, 36-84 hours for weather).

Scoring recipe

def compute_skill_score(predictions, gold, benchmark):
    forecaster_score = quantile_scoring_function(predictions, gold)
    benchmark_score = quantile_scoring_function(benchmark, gold)
    skill = benchmark_score - forecaster_score
    return skill

Common pitfalls

• Evaluating on a static train/test split instead of the required real-time sequential submission process.
• Confusing calibration (coverage rates) with sharpness (interval width); both are needed for good probabilistic forecasts.
• Small sample sizes per horizon (13-14 weeks) can lead to noisy coverage estimates.

Evidence (verbatim from paper)

Figure 3 summarizes performance as measured by the quantile scoring function, based on which we compute skill scores relative to the benchmark forecasts (see Equation 1). A positive skill score indicates that a forecast outperforms the benchmark.

Citation

@misc{bracher2022learning,
  title={Learning to forecast: The probabilistic time series forecasting challenge},
  author={Bracher et al. (2022)},
  year={2022},
  note={arXiv:2211.16171}
}

• arXiv: 2211.16171