India Weather Bench Eval

india-weather-bench-eval

IndiaWeatherBench: A Dataset and Benchmark for Data-Driven Regional Weather Forecasting over India — Nguyen et al. (2025) (arXiv:2509.00653, 2025)

What this evaluates

Evaluates data-driven regional weather forecasting models over India under varying boundary conditioning strategies. It probes the ability of architectures to accurately predict multi-variable meteorological fields at high resolution and assesses their robustness during extreme weather events like heatwaves.

Datasets

• IndiaWeatherBench — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/tung-nd/IndiaWeatherBench

Metrics

• RMSE (primary) — range: other
  • Root Mean Squared Error computed over the spatial grid (256x256), meteorological variables, and forecast lead times. Lower values indicate better accuracy.

Input / output format

Input: A 39-channel tensor of shape (256, 256, 39) containing regional meteorological variables (e.g., 2m temperature, 10m wind, MSLP, 7 pressure levels). Optionally concatenated with boundary forcing (10-pixel ring of ground truth) or coarse-resolution ERA5 conditioning (124x124 interpolated to 256x256).

Output: A predicted 39-channel tensor of shape (256, 256, 39) representing the regional weather state at the target forecast time step.

Scoring recipe

def compute_rmse(pred, gold):
    # pred, gold: (batch, 256, 256, 39)
    diff = pred - gold
    mse = np.mean(diff ** 2, axis=(0, 1, 2, 3))
    return np.sqrt(mse).mean()

Common pitfalls

• Architectures may fail when boundary conditioning strategy mismatches their input tokenization (e.g., Stormer degrades with coarse-resolution conditioning due to scale mixing in patches).
• Deterministic RMSE alone may mask systematic biases during extreme events; models can show low average error but large warm/cold biases in heatwaves.

Evidence (verbatim from paper)

We use RMSE as the evaluation metric, and refer readers to Appendix[10.2] for additional metrics.

Citation

@misc{nguyen2025indiaweatherbench,
  title={IndiaWeatherBench: A Dataset and Benchmark for Data-Driven Regional Weather Forecasting over India},
  author={Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2509.00653}
}

• arXiv: 2509.00653