Uk Windstorm Generation Eval

uk-windstorm-generation-eval

Using Generative Models to Produce Realistic Populations of the United Kingdom Windstorms — Tsoi et al. (2024) (arXiv:2409.10696, 2024)

What this evaluates

Evaluates the ability of generative models to synthesize realistic 10-meter hourly wind speed maps over the UK, focusing on statistical fidelity, spatial structure, and extreme event intensity distribution.

Datasets

• ERA5 — total ?; splits: test (-1)

Metrics

• FID (primary) — range: [0, inf)
  • Fréchet Inception Distance; measures the distance between feature distributions of real and generated samples using their mean and covariance vectors. Lower values indicate better distributional match.
• SSIM — range: [0, 1]
  • Structural Similarity Index Measure computed on the average storm severity index (SSI) map; evaluates spatial structure and intensity preservation between real and generated average maps.
• KL divergence — range: [0, inf)
  • Mean Kullback-Leibler divergence between the probability distributions of real and generated samples projected onto the first 25 principal component dimensions.
• EMD — range: [0, inf)
  • Earth Mover's Distance on the first 25 principal component dimensions; quantifies the minimum work required to transform the generated distribution into the real distribution.

Input / output format

Input: 10-meter hourly wind speed maps over the UK (grid data) from the ERA5 reanalysis dataset.

Output: Synthesized 10-meter hourly wind speed maps over the UK generated by the model.

Scoring recipe

def evaluate(real_maps, gen_maps):
    # Compute FID between real and generated feature distributions
    fid = frechet_distance(real_maps, gen_maps)
    # Compute SSIM on average SSI maps
    avg_real_ssi = compute_ssi(real_maps).mean()
    avg_gen_ssi = compute_ssi(gen_maps).mean()
    ssim = structural_similarity(avg_real_ssi, avg_gen_ssi)
    # Project both sets to first 25 principal components
    real_pc = pca(real_maps, n_components=25)
    gen_pc = pca(gen_maps, n_components=25)
    # Compute distributional metrics on PC space
    kl = kullback_leibler_divergence(real_pc, gen_pc)
    emd = earth_movers_distance(real_pc, gen_pc)
    return {'FID': fid, 'SSIM': ssim, 'KL divergence': kl, 'EMD': emd}

Common pitfalls

• SSIM is computed on the average SSI map rather than individual wind speed maps, which may mask spatial inaccuracies in specific samples.
• KL divergence and EMD are calculated on the first 25 PC dimensions, not the full high-dimensional grid, potentially overlooking fine-grained spatial details.
• Extreme event evaluation relies on top-10/100 SSI cases, which may not represent the full tail distribution of windstorm intensities.

Evidence (verbatim from paper)

The performance summary table (Table 1) compares the generative models across four different metrics. The WGAN-GP model consistently performs the best across most metrics, achieving the lowest FID, KL divergence, and EMD. This suggests that the WGAN-GP model produces images that are closest to the real dataset in terms of overall quality, variability, and distribution.

Citation

@misc{tsoi2024windstorms,
  title={Using Generative Models to Produce Realistic Populations of the United Kingdom Windstorms},
  author={Tsoi et al. (2024)},
  year={2024},
  note={arXiv:2409.10696}
}

• arXiv: 2409.10696