Climax Letkf Eval

climax-letkf-eval

Ensemble data assimilation to diagnose AI-based weather prediction model: A case with ClimaX version 0.3.1 — Kotsuki et al. (2024) (arXiv:2407.17781, 2024)

What this evaluates

Evaluates the stability and error covariance representation of an AI-based weather prediction model (ClimaX) when integrated into an ensemble data assimilation system (LETKF). It probes the model's ability to generate physically consistent ensemble forecasts, capture flow-dependent error growth, and propagate observation information to unobserved variables without filter divergence.

Datasets

• WeatherBench — total ?; splits: test (-1)

Metrics

• RMSE (primary) — range: other
  • Global-mean root mean square error computed over time for specific atmospheric variables (temperature, geopotential height, winds, humidity, surface pressure). Lower values indicate better forecast/analysis accuracy.
• MAE_diff — range: other
  • Mean absolute error difference between analysis and first-guess states relative to WeatherBench truth: MAE_diff = (1/N_t) Σ_t (|x_t^a - x_t^WB| - |x_t^b - x_t^WB|). Negative values indicate DA improvement, positive values indicate degradation.
• Inflation factor (β) — range: other
  • Covariance inflation factor estimated from observation-space statistics during the LETKF cycle. Used to diagnose error growth rates and chaotic behavior of the AI model.

Input / output format

Input: Ensemble forecast states from ClimaX, observation data, and localization scale parameters (L_h) for the LETKF data assimilation cycle.

Output: Analysis states, first-guess (background) states, and computed error metrics (RMSE, MAE_diff) for each variable and localization scale configuration.

Scoring recipe

def evaluate(analysis_states, first_guess_states, truth_states):
    rmse = {}
    mae_diff = {}
    for var in variables:
        pred = analysis_states[var]
        gold = truth_states[var]
        bg = first_guess_states[var]
        rmse[var] = np.sqrt(np.mean((pred - gold)**2))
        mae_diff[var] = np.mean(np.abs(pred - gold)) - np.mean(np.abs(bg - gold))
    return rmse, mae_diff

Common pitfalls

• Filter divergence at large localization scales (e.g., 800 km) stems from erroneous error covariance, not model failure.
• Improvements in observed variables (temperature) do not guarantee improvements in unobserved variables (geopotential height) or winds due to suboptimal spatial/inter-variable covariance.
• The inflation factor β diagnoses error growth dynamics rather than direct forecasting accuracy; lower β indicates slower error growth compared to dynamical models.

Evidence (verbatim from paper)

Figure 3 presents the time series of global-mean root mean square errors (RMSEs) for temperature and geopotential height at the fifth model level, with four different horizontal localization scales (L_h). After the initiation of data assimilation, all experiments showed reductions in analysis errors. ... Here, we investigate the spatial patterns of the difference between the analysis and first-guess mean absolute errors, which is given by: MAE_diff = (1/N_t) Σ_t |x_t^a - x_t^WB| - |x_t^b - x_t^WB|, where N_t is the sample size and superscript WB represents WeatherBench data.

Citation

@misc{kotsuki2024ensemble,
  title={Ensemble data assimilation to diagnose AI-based weather prediction model: A case with ClimaX version 0.3.1},
  author={Kotsuki et al. (2024)},
  year={2024},
  note={arXiv:2407.17781}
}

• arXiv: 2407.17781