Dabench Eval

dabench-eval

A Benchmark for AI-based Weather Data Assimilation — Wuxin Wang et al. (2024) (arXiv:2408.11438, 2024)

What this evaluates

This benchmark evaluates AI-based data assimilation models for weather forecasting. It tests their ability to integrate sparse or noisy observations with background atmospheric fields to produce accurate analysis fields, which are then used to initialize skillful medium-range weather predictions.

Datasets

• ERA5 — total ?; splits: train (-1), test (-1)
• GDAS — total ?; splits: test (-1)

Metrics

• RMSE (primary) — range: other
  • Root Mean Square Error. Computed as the square root of the mean squared difference between predicted and ground-truth fields.
• Bias — range: other
  • Mean error. Computed as the average difference between predicted and ground-truth fields.
• ACC — range: [-1, 1]
  • Anomaly Correlation Coefficient. Measures the correlation between predicted and observed anomalies relative to climatology.
• Activity — range: other
  • Measures the variability or extreme values of the atmospheric field, typically calculated as the standard deviation or variance of the predicted field.

Input / output format

Input: Background atmospheric field (e.g., from a 48-hour forecast or reanalysis) combined with sparse observations (simulated via random grid masking or real-world GDAS sounding data).

Output: Assimilated analysis field representing the corrected atmospheric state, used as the initial condition for the next 12-hour forecast cycle.

Scoring recipe

def compute_metrics(predictions, gold, climatology=None):
    rmse = np.sqrt(np.mean((predictions - gold) ** 2))
    bias = np.mean(predictions - gold)
    if climatology is not None:
        acc = np.corrcoef((predictions - climatology).flatten(), (gold - climatology).flatten())[0, 1]
    else:
        acc = np.corrcoef(predictions.flatten(), gold.flatten())[0, 1]
    activity = np.std(predictions)
    return {'RMSE': rmse, 'Bias': bias, 'ACC': acc, 'Activity': activity}

Common pitfalls

• Using climatology as an initial field yields low RMSE but near-zero ACC, misleadingly suggesting good performance while actually producing unreliable forecasts.
• Evaluation results are highly sensitive to observation sparsity; models trained at 90% masking degrade sharply at 95% or 99%, requiring explicit reporting of mask ratios.
• Daily metrics are frequently smoothed using an 11-point exponential moving average (EMA) for visualization, which can obscure short-term instability in the raw scores.

Evidence (verbatim from paper)

We compute the root mean square error (RMSE) and Bias for the analysis fields generated by deterministic DA. For ensemble DA, we further assess the Continuous Ranked Probability Score (CRPS) and the Spread-Skill Ratio (SSR). In medium-range weather forecasting experiments, we evaluate the RMSE, the anomaly correlation coefficient (ACC), and the Activity of 10-day predictions initialized from the aforementioned analysis fields.

Citation

@misc{wang2024dabench,
  title={A Benchmark for AI-based Weather Data Assimilation},
  author={Wuxin Wang et al. (2024)},
  year={2024},
  note={arXiv:2408.11438}
}

• arXiv: 2408.11438