Metnet 3 Weather Forecast Eval

metnet-3-weather-forecast-eval

Deep Learning for Day Forecasts from Sparse Observations — Andrychowicz et al. (2023) (arXiv:2306.06079, 2023)

What this evaluates

Evaluates a neural weather model's ability to generate high-resolution, fully dense forecasts of precipitation and surface variables over CONUS from sparse observational data, extending lead times up to 24 hours.

Datasets

• MRMS & OMO Weather Network — total ?; splits: train (-1), test (-1)

Metrics

• CRPS (primary) — range: other
  • Continuous Ranked Probability Score. Measures the accuracy of the full forecast probability distribution against the observed value. Lower is better.
• CSI — range: [0, 1]
  • Critical Success Index. Ratio of correctly predicted events to the sum of hits, false alarms, and misses after thresholding the probabilistic output. Higher is better.
• MAE — range: other
  • Mean Absolute Error. Average absolute difference between predicted and observed values. Lower is better.

Input / output format

Input: Sparse time-series observations from 942 CONUS weather stations (for surface variables) or radar/gauge precipitation data, provided as spatial grids over CONUS.

Output: Marginal probability distribution for each output variable and spatial location, generated via a full categorical Softmax layer.

Scoring recipe

def compute_metrics(pred_dist, true_val, thresholds):
    # CRPS for categorical distribution
    cdf_pred = np.cumsum(pred_dist.probs)
    crps = np.sum((cdf_pred - (true_val >= pred_dist.bins))**2 * pred_dist.dbin)
    
    # CSI for thresholded probabilities
    pred_binary = (pred_dist.probs >= thresholds['prob']).astype(int)
    true_binary = (true_val >= thresholds['val']).astype(int)
    tp = np.sum(pred_binary & true_binary)
    fp = np.sum(pred_binary & ~true_binary)
    fn = np.sum(~pred_binary & true_binary)
    csi = tp / (tp + fp + fn) if (tp + fp + fn) > 0 else 0.0
    
    # MAE
    mae = np.mean(np.abs(pred_dist.mean - true_val))
    return crps, csi, mae

Common pitfalls

• Evaluating surface variable forecasts on training stations instead of the specified 20% hold-out set, which inflates performance and ignores the densification generalization goal.
• Applying CRPS to deterministic baselines (HRRR, HRES) without accounting for their lack of distributional output; CRPS is designed for probabilistic/ensemble forecasts.
• Using a fixed threshold for CSI instead of the validation-optimized thresholds specified in the protocol.

Evidence (verbatim from paper)

We compare the models' performance based on the metrics Continuous Ranked Probability Score (CRPS), Critical Success Index (CSI) and Mean Absolute Error (MAE). CRPS is particularly appropriate for comparison with ENS and HREF as they are ensembles of respectively 50 and 10 members and measures the accuracy of the full output distribution for all possible rates or amounts.

Citation

@misc{andrychowicz2023deeplearning,
  title={Deep Learning for Day Forecasts from Sparse Observations},
  author={Andrychowicz et al. (2023)},
  year={2023},
  note={arXiv:2306.06079}
}

• arXiv: 2306.06079