Ept15 Weather Bench Eval

ept15-weather-bench-eval

EPT-1.5 Technical Report — Molinaro et al. (2024) (arXiv:2410.15076, 2024)

What this evaluates

Evaluates the accuracy of AI weather forecasting models against established numerical models and ground-truth observations. It probes the model's ability to predict atmospheric variables (e.g., wind speed, solar radiation) at hourly resolution over 20-day lead times.

Datasets

• ERA5 — total ?; splits: test (-1)
• IFS HRES IC — total ?; splits: test (-1)
• Weather Stations — total ?; splits: test (-1)

Metrics

• Skill Score (SS) (primary) — range: other (typically [-1, 1], unbounded above)
  • SS = 1 - (RMSE_model / RMSE_reference), where RMSE = sqrt(sum(w_i * (X_tilde_i - X_i)^2)). Compares model error against a reference forecast (IFS HRES). Higher values indicate better performance.

Input / output format

Input: Gridded or point-wise initial weather conditions (ERA5 or IFS HRES IC) and reference forecast data.

Output: Predicted gridded or point-wise weather state variables (e.g., wind speed, solar radiation) at specified lead times.

Scoring recipe

def compute_skill_score(predictions, ground_truth, reference, weights):
    rmse_model = np.sqrt(np.sum(weights * (predictions - ground_truth)**2))
    rmse_ref = np.sqrt(np.sum(weights * (reference - ground_truth)**2))
    return 1.0 - (rmse_model / rmse_ref)

Common pitfalls

• Using ERA5 as ground truth for IFS HRES forecasts introduces initial error (RMSE > 0 at t=0), unfairly penalizing the numerical model. The protocol mandates using consistent initial conditions or point observations.
• Weather station sensors are mounted at 2-10m height, not 100m, so direct comparison with model outputs at 100m requires caution or height adjustment.
• The weighting factor w_i in the RMSE formula is not explicitly defined in the text, which could lead to inconsistent aggregation across variables or grid cells.

Evidence (verbatim from paper)

Performance is evaluated based on the skill score (SS) defined as SS=1−RMSE_model/RMSE_reference, with RMSE denoting the Root Mean Square Error: RMSE=√∑{i=1}^{n}w{i}(˜X_{i}−X_{i})^{2}. Here, ˜X_{i} represents the predicted value, X_{i} is the weather state corresponding to the ground truth dataset, while the reference forecast is IFS HRES as outlined above.

Citation

@misc{molinaro2024ept15,
  title={EPT-1.5 Technical Report},
  author={Molinaro et al. (2024)},
  year={2024},
  note={arXiv:2410.15076}
}

• arXiv: 2410.15076