Duq Weather Forecasting Eval

duq-weather-forecasting-eval

Deep Uncertainty Quantification: A Machine Learning Approach for Weather Forecasting — Bin Wang et al. (arXiv:1812.09467, 2018)

What this evaluates

Evaluates a deep learning model's capability to perform spatio-temporal weather forecasting and quantify predictive uncertainty. It tests the model's ability to fuse historical observations with numerical weather prediction (NWP) data to generate accurate point forecasts and reliable 90% prediction intervals over a 37-hour horizon.

Datasets

• Beijing weather dataset — total ?; splits: train (11480), val (870), test (90)

Metrics

• SS_avg (primary) — range: [-1, 1]
  • Average Skill Score across 9 test days, calculated as 1 - (RMSE_model / RMSE_baseline). Measures relative improvement over a baseline forecast.
• RMSE_avg — range: other
  • Average Root Mean Square Error across 9 test days. Computed as the square root of the mean squared difference between predicted and observed values.
• PICP_avg — range: [0, 1]
  • Average Prediction Interval Coverage Probability across 9 test days. Represents the fraction of ground truth values that fall within the predicted 90% interval.

Input / output format

Input: Encoder inputs: historical observations (28 hours × 10 stations × 9 variables), NWP forecasts, and time/station IDs. Decoder inputs: concatenated NWP forecasts and IDs.

Output: Predicted values for 3 target variables (t2m, rh2m, w10m) over 37 forecast hours, along with corresponding 90% prediction intervals.

Scoring recipe

def compute_metrics(preds, targets, intervals, nwp_preds):
    rmse = np.sqrt(np.mean((preds - targets) ** 2))
    rmse_nwp = np.sqrt(np.mean((nwp_preds - targets) ** 2))
    ss = 1 - (rmse / rmse_nwp)
    picp = np.mean((targets >= intervals[:, 0]) & (targets <= intervals[:, 1]))
    return ss, rmse, picp

Common pitfalls

• SS and RMSE rankings can diverge because they are not linearly related; a model optimizing one may not optimize the other.
• The test set is extremely small (9 days), so statistical significance relies on a one-tail paired T-test with a relaxed significance level of 0.25.
• Prediction interval widths do not monotonically increase over the forecast horizon due to diurnal weather patterns (narrower at night, wider during daytime fluctuations).

Evidence (verbatim from paper)

We also evaluated all methods by $RMSE_{avg}$ as shown in Table [2]. Since $RMSE_{avg}$ and $SS_{avg}$ do not have a fully linear relationship, the counterpart assessment does not reach the optimum at the same time while DUQEsb10 still achieves the best $RMSE_{avg}$.

Citation

@misc{wang2018deepuncertainty,
  title={Deep Uncertainty Quantification: A Machine Learning Approach for Weather Forecasting},
  author={Bin Wang et al.},
  year={2018},
  note={arXiv:1812.09467}
}

• arXiv: 1812.09467