Uk Weather Da Eval

uk-weather-da-eval

Data Assimilation using ERA5, ASOS, and the U-STN model for Weather Forecasting over the UK — Wenqi Wang et al. (2024) (arXiv:2401.07604, 2024)

What this evaluates

Evaluates the impact of data assimilation (DA) using the SPEnKF algorithm on a U-STN12 deep learning model for UK temperature forecasting. It probes the model's ability to integrate global atmospheric data (ERA5 T850) and surface observations (ASOS/ERA5 T2m) over a 120-hour lead time, measuring forecast accuracy degradation or improvement under varying noise levels and assimilation frequencies.

Datasets

• ERA5 — total ?; splits: test (-1)
• ASOS — total ?; splits: test (-1)

Metrics

• RMSE (primary) — range: other
  • Root Mean Square Error: RMSE = sqrt(mean((predicted - observed)^2)). Computed between the model's predicted mean and the noisy/synthetic observation data in the T850 domain over a 120-hour lead time.

Input / output format

Input: Initial state derived from a noisy observation (ERA5 T850 with superimposed Gaussian noise or synthetic data). Assimilation of external data (synthetic, ASOS, or ERA5 T2m) occurs every 24 hours using SPEnKF.

Output: Predicted mean temperature field over a 120-hour lead time.

Scoring recipe

def compute_rmse(predictions, observations):
    # predictions, observations: numpy arrays of shape (time_steps, spatial_grid)
    squared_diff = (predictions - observations) ** 2
    mse = np.mean(squared_diff)
    return np.sqrt(mse)

Common pitfalls

• The initial 12-hour RMSE increase is due to lack of training, not DA failure.
• Assimilating surface T2m data (ASOS or ERA5) degrades performance due to interpolation errors and distributional mismatch with atmospheric T850 data, contrary to expectations that more data always helps.
• Evaluations are averaged over 50 random conditions, not single deterministic runs.

Evidence (verbatim from paper)

We then analysed the root mean square error (RMSE) between the predicted mean and the noisy data in the T850 domain over the initial 120 hours across 50 random conditions.

Citation

@misc{wang2024dataassimilation,
  title={Data Assimilation using ERA5, ASOS, and the U-STN model for Weather Forecasting over the UK},
  author={Wenqi Wang et al. (2024)},
  year={2024},
  note={arXiv:2401.07604}
}

• arXiv: 2401.07604