Fuxi 2 Weather Forecast Eval

fuxi-2-weather-forecast-eval

FuXi-2.0: Advancing machine learning weather forecasting model for practical applications — Zhong et al. (2024) (arXiv:2409.07188, 2024)

What this evaluates

Evaluates the accuracy and temporal consistency of 1-hourly global weather forecasts up to 90 hours lead time. It probes the model's ability to capture both large-scale atmospheric patterns and fine-scale variability across meteorological, energy, aviation, and marine variables.

Datasets

• ERA5 (2018 testing data) — total ?; splits: test (-1)

Metrics

• RMSE (primary) — range: other (variable-dependent)
  • Root mean squared error between the model's forecast and ERA5 observations. Computed globally and latitude-weighted across spatial grids.
• ACC — range: [-1, 1]
  • Anomaly correlation coefficient measuring the Pearson correlation between forecast anomalies and observed anomalies.
• forecast activity — range: other (variable-dependent)
  • Standard deviation of the forecast anomaly, used to assess temporal variability and smoothness of the predictions.

Input / output format

Input: Global atmospheric and oceanic state variables (e.g., T2M, MSL, WS10M, Z500, SST) at 1-hourly temporal resolution and ~0.25° spatial resolution, initialized at specific UTC times.

Output: Predicted global fields for the same variables at 1-hourly resolution for lead times 0 to 90 hours.

Scoring recipe

# For each variable and lead time t:
pred = model.predict(initial_state, t)
obs = ERA5[t]
rmse = np.sqrt(np.mean((pred - obs)**2))
pred_anom = pred - np.mean(pred)
obs_anom = obs - np.mean(obs)
acc = np.corrcoef(pred_anom.flatten(), obs_anom.flatten())[0, 1]
activity = np.std(pred_anom)

Common pitfalls

• Using ECMWF HRES-fc0 as ground truth for 1-hourly evaluation, as its native resolution is 6-hourly and unsuitable for this purpose.
• Optimizing or interpreting RMSE in isolation without checking forecast activity, as lower RMSE can sometimes indicate overly smooth, physically unrealistic forecasts.
• Comparing 1-hourly ML forecasts directly with 6-hourly HRES outputs without accounting for temporal aggregation differences.

Evidence (verbatim from paper)

While RMSE and ACC are critical metrics for assessing forecast quality, the interpretation of model's forecast performance should also consider other forecast characteristics, such as forecast activity, which is defined as standard deviation of the forecast anomaly (see subsection 4.5 for a formal definition).

Citation

@misc{zhong2024fuxi2,
  title={FuXi-2.0: Advancing machine learning weather forecasting model for practical applications},
  author={Zhong et al. (2024)},
  year={2024},
  note={arXiv:2409.07188}
}

• arXiv: 2409.07188