Weather2k Eval

weather2k-eval

Weather2K: A Multivariate Spatio-Temporal Benchmark Dataset for Meteorological Forecasting Based on Real-Time Observation Data from Ground Weather Stations — Zhu et al. (2023) (arXiv:2302.10493, 2023)

What this evaluates

Evaluates the ability of deep learning models to forecast multivariate and univariate meteorological factors (temperature, visibility, humidity) using historical time-series and spatio-temporal data from ground weather stations.

Datasets

• Weather2K — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/bycnfz/weather2k

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error: the average of the absolute differences between predicted and actual values.
• MSE — range: other
  • Mean Squared Error: the average of the squared differences between predicted and actual values.
• RMSE — range: other
  • Root Mean Squared Error: the square root of the mean of the squared differences between predicted and actual values.

Input / output format

Input: Historical time-series data of meteorological factors (20 dimensions for multivariate, 1 for univariate) over a fixed input window (e.g., 72 steps). Spatio-temporal models also receive static geographic coordinates (latitude, longitude, altitude).

Output: Forecasted values for the target meteorological factor(s) over a specified prediction horizon (e.g., 24, 72, 168, 336, or 720 steps).

Scoring recipe

def compute_metrics(predictions, targets):
    mae = np.mean(np.abs(predictions - targets))
    mse = np.mean((predictions - targets) ** 2)
    rmse = np.sqrt(mse)
    return {'MAE': mae, 'MSE': mse, 'RMSE': rmse}

Common pitfalls

• Using multivariate inputs does not guarantee performance gains for univariate forecasting targets; the paper notes it often degrades performance.
• Different baseline categories use different data splits: transformers use a 3:1:2 train/val/test ratio, while classical nonparametric methods use a 3:1 train/test ratio.
• Early stopping patience and learning rate schedules differ between transformer baselines (5 epochs patience, lr 1e-4) and GNN baselines (50 epochs patience, lr 1e-2 with decay).

Evidence (verbatim from paper)

Table 6 shows the performance of multivariate to univariate forecasting. The number in the parenthesis is the difference value from the univariate result, where the better result is highlighted in bold. Table 7: Univariate results of the persistent model and nonparametric methods with different prediction lengths of 24, 72, 168, 336, and 720 steps. The input time length is 72 steps. The reported results of mean and standard deviation are obtained through experiments on Weather2K-S. Metrics | MSE | MAE

Citation

@misc{zhu2023weather2k,
  title={Weather2K: A Multivariate Spatio-Temporal Benchmark Dataset for Meteorological Forecasting Based on Real-Time Observation Data from Ground Weather Stations},
  author={Zhu et al. (2023)},
  year={2023},
  note={arXiv:2302.10493}
}

• arXiv: 2302.10493