qhjqhj00 era5-weatherbench-forecasting-eval
Numerical Weather Forecasting using Convolutional-LSTM with Attention and Context Matcher Mechanisms — Tekin et al. (2021) (arXiv:2102.00696, 2021)
What this evaluates
Spatio-temporal forecasting of atmospheric temperature using deep learning models. It probes the ability to capture long-range spatial-temporal dependencies and predict future weather states from historical multi-feature sequences.
Datasets
- ERA5 Turkey — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/sftekin/ieee_weather
- WeatherBench — total ?; splits: train (-1), val (-1), test (-1)
Metrics
RMSE(primary) — range: other- Root Mean Squared Error: sqrt(mean((y_pred - y_true)^2)). Measures the standard deviation of prediction errors across all grid cells and time steps.
MAE— range: other- Mean Absolute Error: mean(|y_pred - y_true|). Measures the average magnitude of errors without considering direction.
MAPE— range: percent- Mean Absolute Percentage Error: mean(|(y_true - y_pred) / y_true|) * 100. Expresses accuracy as a percentage relative to ground truth values.
ACC— range: [-1, 1]- Anomaly Correlation Coefficient: 1 - sum((y_pred - y_true)^2) / sum((y_true - y_mean)^2). Measures the correlation of forecast anomalies relative to the climatological mean.
Input / output format
Input: Spatio-temporal sequences of weather features (e.g., temperature, wind, humidity) on a 2D grid. Input length is typically 10 time steps (3-hour frequency) or adapted per dataset.
Output: Predicted temperature values for the next N time steps on the same 2D grid. Output length varies by method: 5 steps (sequential ERA5), 72 steps (sequential WeatherBench), 6 steps (iterative WeatherBench), or 3 steps (direct forecast).
Scoring recipe
import numpy as np
def compute_metrics(y_true, y_pred):
rmse = np.sqrt(np.mean((y_pred - y_true)**2))
mae = np.mean(np.abs(y_pred - y_true))
mape = np.mean(np.abs((y_true - y_pred) / y_true)) * 100
y_mean = np.mean(y_true)
acc = 1 - np.sum((y_pred - y_true)**2) / np.sum((y_true - y_mean)**2)
return {'RMSE': rmse, 'MAE': mae, 'MAPE': mape, 'ACC': acc}
Common pitfalls
- Confusion between sequential, iterative, and direct forecasting protocols leads to mismatched output lengths and invalid comparisons.
- Table III contains a copy-paste error where the Weather Model's RMSE/MAE/MAPE values are identical to Table II, likely a typo in the original publication.
- Metrics are computed on raw temperature values, but inputs are min-max normalized per batch, requiring inverse transformation before metric calculation.
Evidence (verbatim from paper)
We split the dataset into the train, validation and test sets, with 0.8, 0.1 and 0.1 split ratios, respectively. ... The numerical results of the experiment are illustrated in Table II. We illustrate the performance of the models on the high-resolution dataset. ... TABLE II: We illustrate the performance of the models on the high-resolution dataset. The best results for each metric are given in bold. | Model Name | RMSE | MAE | MAPE |
Citation
@misc{tekin2021numericalweather,
title={Numerical Weather Forecasting using Convolutional-LSTM with Attention and Context Matcher Mechanisms},
author={Tekin et al. (2021)},
year={2021},
note={arXiv:2102.00696}
}
- arXiv: 2102.00696