Pear Weather Eval

pear-weather-eval

PEAR: Equal Area Weather Forecasting on the Sphere — Linander et al. (2025) (arXiv:2505.17720, 2025)

What this evaluates

Evaluates medium-term weather forecasting capability on a spherical grid by predicting atmospheric variables up to 10 days ahead. It probes the model's ability to capture spatial and temporal dynamics without grid-induced resolution biases.

Datasets

• ERA5-lite — total ?; splits: train (-1), test (-1)

Metrics

• RMSE — range: other
  • Root mean squared error between predicted and ground truth values across all grid cells. Formula: sqrt(1/(12*n_side^2) * sum((y^i - y_hat^i)^2)).
• ACC (primary) — range: other
  • Anomaly correlation coefficient measuring the correlation between deviations from the climatology mean of predicted and ground truth forecasts. Formula: sum(delta_y * delta_y_hat) / sqrt(sum(delta_y^2) * sum(delta_y_hat^2)).

Input / output format

Input: Multi-channel tensor of atmospheric variables (surface and 13 upper-air levels) on a spherical grid (HEALPix n_side=64 or Driscoll-Healy).

Output: Multi-channel tensor of predicted atmospheric variables at specified future lead times (1, 3, 5, or up to 10 days).

Scoring recipe

def compute_metrics(y_true, y_pred, climatology):
    delta_y = y_true - climatology
    delta_y_hat = y_pred - climatology
    rmse = np.sqrt(np.mean((y_true - y_pred) ** 2))
    acc = np.sum(delta_y * delta_y_hat) / np.sqrt(np.sum(delta_y ** 2) * np.sum(delta_y_hat ** 2))
    return rmse, acc

Common pitfalls

• Forgetting to subtract the climatology mean before computing ACC, which inflates correlation due to seasonal cycles.
• Applying latitude-based weighting to HEALPix grids, which is unnecessary because HEALPix cells are equal-area.
• Comparing RMSE across variables without noting that lower is better, while higher ACC is better.

Evidence (verbatim from paper)

At each lead time we calculate the average RMSE and anomaly correlation coefficient (ACC) [37] of all variables over the globe according to

$$ \operatorname {R M S E} (y, \hat {y}) = \sqrt {\frac {1}{1 2 n _ {\text {s i d e}} ^ {2}} \sum_ {i = 0} ^ {1 2 n _ {\text {s i d e}} ^ {2}} \left(y ^ {i} - \hat {y} ^ {i}\right) ^ {2}} \tag {2} $$

$$ \operatorname {A C C} (y, \hat {y}) = \frac {\sum_ {i} ^ {1 2 n _ {\mathrm {n s i d e}} ^ {2}} \Delta y ^ {i} \Delta \hat {y} ^ {i}}{\sqrt {\left(\sum_ {i} ^ {1 2 n _ {\mathrm {n s i d e}} ^ {2}} (\Delta y ^ {i}) ^ {2}\right) \left(\sum_ {i} ^ {1 2 n _ {\mathrm {n s i d e}} ^ {2}} (\Delta \hat {y} ^ {i}) ^ {2}\right)}}, \tag {3} $$

where $\Delta y$ is the difference between the predictions and the climatology average. To evaluate baseline-predictions on Driscoll-Healy, we apply the latitude weighting used in prior work [1]. The equal area grid cells of HEALPix make this reweighting redundant for PEAR. The ACC measures the correlation between deviations from the climatology mean of predicted and ground truth forecasts, with a value of 1 indicating perfect agreement [7]. The climatology average is subtracted to factor out seasonal variations

Citation

@misc{linander2025pear,
  title={PEAR: Equal Area Weather Forecasting on the Sphere},
  author={Linander et al. (2025)},
  year={2025},
  note={arXiv:2505.17720}
}

• arXiv: 2505.17720