Fengwu Weather Forecast Eval

fengwu-weather-forecast-eval

FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead — Kang Chen et al. (arXiv:2304.02948, 2023)

What this evaluates

Evaluates global medium-range weather forecasting accuracy over 10-day lead times across multiple atmospheric variables. It measures point-wise prediction error and anomaly correlation against observed climatology, with explicit latitude weighting to account for spherical grid distortion.

Datasets

• ERA5 — total ?; splits: test (-1)

Metrics

• RMSE — range: other
  • Latitude-weighted Root Mean Square Error averaged over T test time slots. For each slot, computes the square root of the mean squared error across the grid, weighted by the cosine of latitude normalized by the sum of cosines.
• ACC (primary) — range: other
  • Latitude-weighted Anomaly Correlation Coefficient averaged over T test time slots. Computes the ratio of latitude-weighted covariance between prediction anomalies and target anomalies to the product of their latitude-weighted standard deviations.

Input / output format

Input: Initial atmospheric state fields (e.g., temperature, geopotential height) at 00z or 12z for a given day, represented as multi-channel grid data.

Output: Predicted atmospheric state fields for lead times up to 10 days at 6-hour intervals, matching the input grid resolution and channels.

Scoring recipe

def compute_metrics(preds, targets, climatology, latitudes):
    weights = np.cos(np.radians(latitudes))
    weights = weights / np.sum(weights, axis=(0,1), keepdims=True)
    anomalies_pred = preds - climatology
    anomalies_target = targets - climatology
    rmse_scores, acc_scores = [], []
    for t in range(T):
        err = weights * (anomalies_target[t] - anomalies_pred[t])**2
        rmse_scores.append(np.sqrt(np.sum(err)))
        cov = np.sum(weights * anomalies_pred[t] * anomalies_target[t])
        std_p = np.sqrt(np.sum(weights * anomalies_pred[t]**2))
        std_t = np.sqrt(np.sum(weights * anomalies_target[t]**2))
        acc_scores.append(cov / (std_p * std_t + 1e-8))
    return np.mean(rmse_scores), np.mean(acc_scores)

Common pitfalls

• Using hourly climatology instead of daily climatology for ACC calculation significantly inflates scores due to outlier sensitivity.
• Omitting the latitude weighting factor (cosine of latitude) misrepresents model skill at high latitudes versus the equator.
• Evaluating on lead times or initial conditions different from the standard 00z/12z 10-day protocol breaks comparability with GraphCast.

Evidence (verbatim from paper)

For consistency, we follow the evaluation protocols implemented in the work by GraphCast, which includes the same evaluation metrics, dataset splitting, and lead time of forecast. With 00z and 12z as the initial weather states for each day, we compare the performance of FengWu and GraphCast for a 10-day forecast using the commonly used RMSE and ACC metrics based on the test set.

Citation

@misc{chen2023fengwu,
  title={FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead},
  author={Kang Chen et al.},
  year={2023},
  note={arXiv:2304.02948}
}

• arXiv: 2304.02948