Subseasonal Climate Usa Eval

subseasonal-climate-usa-eval

SubseasonalClimateUSA: A Dataset for Subseasonal Forecasting and Benchmarking — Mouatadid et al. (2021) (arXiv:2109.10399, 2021)

What this evaluates

Evaluates the ability of machine learning and dynamical models to forecast subseasonal temperature and precipitation over the contiguous United States at 3–4 and 5–6 week lead times. It benchmarks predictive accuracy against operational baselines and tests robustness to high noise and spatial heterogeneity in climate data.

Datasets

• SubseasonalClimateUSA — total ?; splits: test (-1); repo https://github.com/microsoft/subseasonal_data

Metrics

• % IMPROVEMENT OVER MEAN DEB. CFSV2 RMSE (primary) — range: percent
  • Calculated as (RMSE_baseline - RMSE_model) / RMSE_baseline * 100. Measures the relative reduction in root mean squared error compared to the debiased CFSv2 baseline.
• AVERAGE % SKILL — range: percent
  • Identical to RMSE improvement in this protocol: (RMSE_baseline - RMSE_model) / RMSE_baseline * 100. Used interchangeably with RMSE improvement to report model skill relative to the baseline.

Input / output format

Input: Spatiotemporal ground truth observations and operational model forecasts (e.g., CFSv2, ECMWF) for the contiguous U.S., typically on a 1.5°×1.5° or 1°×1° grid.

Output: Predicted temperature and precipitation values for lead times of 3–4 weeks and 5–6 weeks, issued weekly.

Scoring recipe

def compute_metrics(preds, gold, baseline):
    rmse_pred = np.sqrt(np.mean((preds - gold)**2))
    rmse_base = np.sqrt(np.mean((baseline - gold)**2))
    rmse_improvement = ((rmse_base - rmse_pred) / rmse_base) * 100
    skill = ((rmse_base - rmse_pred) / rmse_base) * 100
    return rmse_improvement, skill

Common pitfalls

• Dynamical model forecasts must be debiased before fair comparison, otherwise RMSE/skill scores are artificially penalized.
• Complex deep learning models often overfit due to high noise levels in subseasonal data, making simpler hybrid or baseline models more robust.
• Performance is highly heterogeneous across space and time; averaging over the entire contiguous U.S. can mask regional strengths or weaknesses.

Evidence (verbatim from paper)

We generate forecasts for each Wednesday in the years 2011-2020 and, for each reported period, we assess both mean RMSE relative to a baseline model and average percentage skill.

Citation

@misc{mouatadid2021subseasonalclimateusa,
  title={SubseasonalClimateUSA: A Dataset for Subseasonal Forecasting and Benchmarking},
  author={Mouatadid et al. (2021)},
  year={2021},
  note={arXiv:2109.10399}
}

• arXiv: 2109.10399