Puyun Weather Forecasting Eval

puyun-weather-forecasting-eval

PuYun: Medium-Range Global Weather Forecasting Using Large Kernel Attention Convolutional Networks — Zhu et al. (2024) (arXiv:2409.02123, 2024)

What this evaluates

Evaluates medium-range global weather forecasting capability using an autoregressive convolutional network. It measures prediction accuracy over 10-day horizons at 6-hour intervals against reanalysis ground truth.

Datasets

• ERA5 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error averaged across all channels, latitudes, and longitudes, optionally weighted by latitude.
• RMSE — range: other
  • Root Mean Square Error computed per variable and grid point, typically averaged over the test set. Commonly reported for Z500 and T2M.

Input / output format

Input: Tensor of shape (2, 69, 721, 1440) representing two preceding 6-hourly time steps, 69 atmospheric/surface variables, and global grid points (721 latitudes × 1440 longitudes).

Output: Predicted tensor of shape (1, 69, 721, 1440) for the next 6-hour time step. Autoregressively chained for multi-step forecasts.

Scoring recipe

def compute_mae(pred, target, lat_weights):
    # pred, target: (C, H, W)
    # lat_weights: (H,) latitude scaling factor
    weighted_diff = lat_weights * np.abs(pred - target)
    return np.mean(weighted_diff)

Common pitfalls

• Autoregressive error accumulation over long horizons can degrade accuracy significantly.
• Latitude-weighted loss during training does not directly translate to unweighted evaluation metrics like RMSE.
• Precipitation variables were excluded during training but may be relevant for full atmospheric assessment.

Evidence (verbatim from paper)

To train and test the PuYun model, we utilize 40 years of historical weather data (1979-2018) from ERA5 reanalysis archive. Specifically, data from 1979 to 2016 is used for training, data from 2017 for validation, and data from 2018 for testing. The specific loss function utilized for single-step training is mean absolute error (MAE), defined as:

Citation

@misc{zhu2024puyun,
  title={PuYun: Medium-Range Global Weather Forecasting Using Large Kernel Attention Convolutional Networks},
  author={Zhu et al. (2024)},
  year={2024},
  note={arXiv:2409.02123}
}

• arXiv: 2409.02123