Pinball Score

pinball-score

Probabilistic Forecasting of Regional Net-load with Conditional Extremes and Gridded NWP — Browell & Fasiolo (2021) (arXiv:2103.10335, 2021)

What this evaluates

Evaluates the sharpness and calibration of probabilistic net-load forecasts. It measures how closely predicted quantiles align with actual observations and how narrow the prediction intervals are while maintaining statistical reliability.

Datasets

• Regional net-load dataset — total ?; splits: test (-1)

Metrics

• Pinball Score (primary) — range: other
  • Average loss over a set of quantiles A: (1/(T|A|)) * sum_{alpha in A} sum_{t=1}^T (q_{alpha,t} - y_t) * (1(y_t <= q_{alpha,t}) - alpha). Matches the loss function minimised in quantile regression estimation.
• Interval Width — range: other
  • Average width of the (1-2lambda)% prediction interval, calculated as mean(q_{1-lambda,t} - q_{lambda,t}). Measures sharpness.
• Calibration — range: percent
  • Property that forecast probabilities match observed frequencies (e.g., 10% of observations fall below the alpha=0.1 quantile). Evaluated via reliability diagrams and worm plots with consistency intervals accounting for temporal correlation.

Input / output format

Input: Historical net-load time series observations and gridded Numerical Weather Prediction (NWP) covariates.

Output: Predictive quantiles q_{alpha,t} for a specified set of quantiles A, or prediction intervals.

Scoring recipe

def compute_pinball_score(y_true, q_pred, alphas):
    total_loss = 0.0
    for alpha in alphas:
        q = q_pred[alpha]
        indicator = (y_true <= q).astype(float)
        loss = np.mean((q - y_true) * (indicator - alpha))
        total_loss += loss
    return total_loss / len(alphas)

Common pitfalls

• Tail quantile scores suffer from high variance due to sparse observations, making them unreliable for discriminating between forecasting systems.
• Calibration must be verified separately from sharpness; narrow intervals are useless if not statistically reliable.
• Standard independence assumptions fail for net-load data, requiring consistency intervals that explicitly account for temporal correlation.

Evidence (verbatim from paper)

The Pinball Score for an individual quantile matches the loss function minimised in quantile regression model estimation. The Pinball Score is given by [formula] where A is the set of quantiles being estimated. A drawback of the Pinball Score is that it places greater weight on the performance of quantiles close to alpha=0.5 and less on those in the tails of predictive distributions.

Citation

@misc{browell2021probabilistic,
  title={Probabilistic Forecasting of Regional Net-load with Conditional Extremes and Gridded NWP},
  author={Browell & Fasiolo (2021)},
  year={2021},
  note={arXiv:2103.10335}
}

• arXiv: 2103.10335