# Climax Letkf Eval

> Evaluates the stability and error covariance representation of an AI-based weather prediction model (ClimaX) when integrated into an ensemble data assimilation system (LETKF). It probes the model's ability to generate physically consistent ensemble forecasts, capture flow-dependent error growth, and propagate observation information to unobserved variables without filter divergence. Use when the user wants to benchmark on WeatherBench, or asks about evaluating this task. Reports RMSE.

- Skill: `qhjqhj00/climax-letkf-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/climax-letkf-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/climax-letkf-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Marketing & Growth
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/climax-letkf-eval

---


# climax-letkf-eval

> Ensemble data assimilation to diagnose AI-based weather prediction model: A case with ClimaX version 0.3.1 — Kotsuki et al. (2024) (arXiv:2407.17781, 2024)

## What this evaluates

Evaluates the stability and error covariance representation of an AI-based weather prediction model (ClimaX) when integrated into an ensemble data assimilation system (LETKF). It probes the model's ability to generate physically consistent ensemble forecasts, capture flow-dependent error growth, and propagate observation information to unobserved variables without filter divergence.

## Datasets

- **WeatherBench** — total ?; splits: test (-1)

## Metrics

- `RMSE` **(primary)** — range: other
  - Global-mean root mean square error computed over time for specific atmospheric variables (temperature, geopotential height, winds, humidity, surface pressure). Lower values indicate better forecast/analysis accuracy.
- `MAE_diff` — range: other
  - Mean absolute error difference between analysis and first-guess states relative to WeatherBench truth: MAE_diff = (1/N_t) Σ_t (|x_t^a - x_t^WB| - |x_t^b - x_t^WB|). Negative values indicate DA improvement, positive values indicate degradation.
- `Inflation factor (β)` — range: other
  - Covariance inflation factor estimated from observation-space statistics during the LETKF cycle. Used to diagnose error growth rates and chaotic behavior of the AI model.

## Input / output format

**Input**: Ensemble forecast states from ClimaX, observation data, and localization scale parameters (L_h) for the LETKF data assimilation cycle.

**Output**: Analysis states, first-guess (background) states, and computed error metrics (RMSE, MAE_diff) for each variable and localization scale configuration.

## Scoring recipe

```python
def evaluate(analysis_states, first_guess_states, truth_states):
    rmse = {}
    mae_diff = {}
    for var in variables:
        pred = analysis_states[var]
        gold = truth_states[var]
        bg = first_guess_states[var]
        rmse[var] = np.sqrt(np.mean((pred - gold)**2))
        mae_diff[var] = np.mean(np.abs(pred - gold)) - np.mean(np.abs(bg - gold))
    return rmse, mae_diff
```

## Common pitfalls

- Filter divergence at large localization scales (e.g., 800 km) stems from erroneous error covariance, not model failure.
- Improvements in observed variables (temperature) do not guarantee improvements in unobserved variables (geopotential height) or winds due to suboptimal spatial/inter-variable covariance.
- The inflation factor β diagnoses error growth dynamics rather than direct forecasting accuracy; lower β indicates slower error growth compared to dynamical models.

## Evidence (verbatim from paper)

> Figure 3 presents the time series of global-mean root mean square errors (RMSEs) for temperature and geopotential height at the fifth model level, with four different horizontal localization scales (L_h). After the initiation of data assimilation, all experiments showed reductions in analysis errors. ... Here, we investigate the spatial patterns of the difference between the analysis and first-guess mean absolute errors, which is given by: MAE_diff = (1/N_t) Σ_t |x_t^a - x_t^WB| - |x_t^b - x_t^WB|, where N_t is the sample size and superscript WB represents WeatherBench data.

## Citation

```bibtex
@misc{kotsuki2024ensemble,
  title={Ensemble data assimilation to diagnose AI-based weather prediction model: A case with ClimaX version 0.3.1},
  author={Kotsuki et al. (2024)},
  year={2024},
  note={arXiv:2407.17781}
}
```

- arXiv: 2407.17781