Weather 5k Eval

weather-5k-eval

How far are today's time-series models from real-world weather forecasting applications? — Han et al. (2024) (arXiv:2406.14399, 2024)

What this evaluates

Evaluates the capability of data-driven time-series forecasting models to predict global meteorological variables over short to long horizons, and assesses their robustness in forecasting extreme weather events compared to numerical weather prediction baselines.

Datasets

• WEATHER-5K — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/taohan10200/WEATHER-5K

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error; computes the average of absolute differences between predicted and observed values across all stations and time steps.
• MSE — range: other
  • Mean Square Error; computes the average of squared differences between predicted and observed values, amplifying outlier errors.
• SEDI — range: [0, 1]
  • Symmetric Extremal Dependence Index; calculates the ratio of correctly identified extreme events (both upper and lower quantile thresholds) to the total number of observed extreme events. SEDI ∈ [0,1].

Input / output format

Input: Spatial-temporal time-series tensor X ∈ R^(N×T×V) representing N weather stations, T=48 historical time steps, and V meteorological variables.

Output: Predicted time-series tensor X̂ ∈ R^(N×τ×V) for forecast horizon τ ∈ {24, 72, 120, 168} steps.

Scoring recipe

def compute_metrics(pred, true):
    mae = np.mean(np.abs(pred - true))
    mse = np.mean((pred - true) ** 2)
    sedis = {}
    for var in V:
        for p in [0.90, 0.95, 0.98, 0.995, 0.10, 0.05, 0.02, 0.005]:
            q_lower = np.percentile(true[:, :, var], p * 100)
            q_upper = np.percentile(true[:, :, var], (1 - p) * 100)
            tp = np.sum((pred < q_lower) & (true < q_lower)) + np.sum((pred > q_upper) & (true > q_upper))
            fn = np.sum(true < q_lower) + np.sum(true > q_upper)
            sedis[(var, p)] = tp / fn if fn > 0 else 0.0
    return mae, mse, sedis

Common pitfalls

• All baselines must use a fixed input length of 48 historical steps to ensure fair comparison, regardless of their original implementations.
• Results are reported from a single training run per model rather than averaged over multiple random seeds, relying on dataset stability.
• SEDI thresholds are computed station-specifically based on observed quantiles, not global dataset percentiles.

Evidence (verbatim from paper)

The WEATHER-5K dataset is divided into three subsets: training (with years 2014-2021), validation (with yea 2022), and test (with year 2023), which follows an 8:1:1 ratio. ... Mean Absolute Error (MAE) and Mean Square Error (MSE) are used to evaluate the overall performance of the GSWF. ... SEDI\in[0,1] quantifies the model’s ability to correctly identify extreme weather events.

Citation

@misc{han2024weather5k,
  title={How far are today's time-series models from real-world weather forecasting applications?},
  author={Han et al. (2024)},
  year={2024},
  note={arXiv:2406.14399}
}

• arXiv: 2406.14399