Ens 10 Eval

ens-10-eval

ENS-10: A Dataset For Post-Processing Ensemble Weather Forecasts — Ashkboos et al. (2022) (arXiv:2206.14786, 2022)

What this evaluates

Probes the ability of deep learning and statistical models to correct biases in long-term ensemble weather forecasts. It evaluates how well models can post-process raw ensemble members to produce calibrated predictive distributions for surface and atmospheric variables.

Datasets

• ENS-10 — total ?; splits: test (-1)

Metrics

• CRPS (primary) — range: other (lower is better)
  • Continuous Ranked Probability Score. Assumes a Gaussian predictive distribution N(μ, σ²). CRPS(F, x) = σ[2ψ((x-μ)/σ) + (x-μ)/σ(2φ((x-μ)/σ)-1) - 1/√π], where ψ and φ are the standard Gaussian PDF and CDF, μ and σ are predicted mean and std, and x is the observation.
• EECRPS — range: other (lower is better)
  • Extreme Event-weighted CRPS. Computes CRPS for each grid point, weights it by the absolute value of the Extreme Forecast Index (EFI), and averages over the global test set.

Input / output format

Input: Ensemble forecast members (5 or 10) for multiple weather variables at T=48h, optionally including spatial coordinates (lat/lon or xyz). Data is normalized per model (e.g., time-ensemble mean/std per grid point for MLP/U-Net, or global min/max for EMOS).

Output: Corrected mean (μ) and standard deviation (σ) of a Gaussian distribution for the target variable at each grid point. (Transformer outputs corrected ensemble members, from which μ and σ are computed offline).

Scoring recipe

def compute_crps(mu, sigma, x):
    z = (x - mu) / sigma
    return sigma * (2 * norm.pdf(z) + z * (2 * norm.cdf(z) - 1) - 1 / sqrt(pi))

def compute_ee_crps(crps_map, efi_map):
    return np.mean(crps_map * np.abs(efi_map))

# For each grid point in test set:
# 1. Get predicted mu, sigma and observed x
# 2. crps_val = compute_crps(mu, sigma, x)
# 3. For EECRPS, weight crps_val by |EFI_val|
# 4. Average CRPS or weighted CRPS over all grid points

Common pitfalls

• CRPS is non-differentiable, requiring a Gaussian assumption to compute it during training.
• Input normalization strategies vary significantly across baselines (per-grid-point time-ensemble scaling vs. global min-max scaling), complicating direct comparison.
• EMOS performance can degrade with 10 ensemble members due to optimizer sensitivity, requiring careful tuning.

Evidence (verbatim from paper)

We now define several representative baselines for the prediction correction task on ENS-10, and evaluate them all with both CRPS and EECRPS using five and ten ensemble members as input. ... To overcome this issue, following previous work [15, 36, 39], we assume a Gaussian distribution on the target variable and learn the mean and standard deviation of this distribution. In this case, the CRPS from Eqn. (1) can be rewritten as $$ \operatorname {C R P S} \left(F ^ {i, j}, x\right) = \sigma \left[ 2 \psi \left(\frac {x - \mu}{\sigma}\right) + \frac {x - \mu}{\sigma} \left(2 \phi \left(\frac {x - \mu}{\sigma}\right) - 1\right) - \frac {1}{\sqrt {\pi}} \right], $$ ... We use the absolute value of EFI to weight the CRPS metric before averaging over the global data points in our test set.

Citation

@misc{ashkboos2022ens10,
  title={ENS-10: A Dataset For Post-Processing Ensemble Weather Forecasts},
  author={Ashkboos et al. (2022)},
  year={2022},
  note={arXiv:2206.14786}
}

• arXiv: 2206.14786