Climate Downscaling Eval

climate-downscaling-eval

Resolution-Agnostic Transformer-based Climate Downscaling — Curran et al. (2024) (arXiv:2411.14774, 2024)

What this evaluates

Evaluates the ability of deep learning models to perform super-resolution (downscaling) on meteorological surface variables across different spatial resolutions and climate datasets. It probes spatial reconstruction accuracy, structural fidelity, and zero-shot generalization capability in Earth system modeling.

Datasets

• ERA5 — total ?; splits: train (-1), test (-1)
• BARRA-SY — total ?; splits: test (-1)

Metrics

• RMSE (primary) — range: [0, ∞)
  • Root Mean Square Error: the square root of the average of squared differences between predicted and ground truth pixel values. Lower values indicate better accuracy.
• PSNR — range: dB
  • Peak Signal-to-Noise Ratio: 10 * log10(max_val^2 / MSE), where MSE is the mean squared error. Higher values indicate better reconstruction quality.
• SSIM — range: [0, 1]
  • Structural Similarity Index: measures perceived change in structural information, luminance, and contrast between two images. Values range from 0 to 1, with 1 indicating identical structure.

Input / output format

Input: Coarse-resolution 2D climate variable grids (e.g., 0.5° or 3km spacing) for surface variables including 10-meter u/v wind components, 2-meter temperature, and total precipitation.

Output: Fine-resolution 2D climate variable grids (e.g., 0.25° or 1.5km spacing) matching the target 2× downscaling factor.

Scoring recipe

import numpy as np
from skimage.metrics import structural_similarity as ssim

def compute_metrics(pred, true):
    # RMSE
    rmse = np.sqrt(np.mean((pred - true) ** 2))
    # PSNR (assuming normalized data, max_val=1)
    mse = np.mean((pred - true) ** 2)
    psnr = 10 * np.log10(1.0 / mse) if mse > 0 else float('inf')
    # SSIM (single channel, data_range based on actual min/max)
    ssim_val = ssim(pred, true, data_range=pred.max() - pred.min())
    return rmse, psnr, ssim_val

Common pitfalls

• Models are evaluated zero-shot on the target fine resolution without fine-tuning, which differs from standard supervised super-resolution benchmarks.
• Standard image metrics (RMSE/PSNR/SSIM) do not enforce physical conservation laws (e.g., mass conservation), potentially rewarding physically implausible outputs.
• Training data spans differ across baselines (ResNET trained on 2000-2005 vs Earth ViT on 2000-2001), complicating direct performance comparisons.

Evidence (verbatim from paper)

We evaluated the models across all surface variables using three performance metrics: RMSE, PSNR, and SSIM. RMSE measures prediction accuracy, while PSNR (Peak Signal-to-Noise Ratio) assesses the quality of the reconstructed image by comparing the maximum possible signal to the noise. SSIM (Structural Similarity Index) evaluates the structural similarity between predicted and reference images, focusing on aspects like luminance and contrast.

Citation

@misc{curran2024resolutionagnostic,
  title={Resolution-Agnostic Transformer-based Climate Downscaling},
  author={Curran et al. (2024)},
  year={2024},
  note={arXiv:2411.14774}
}

• arXiv: 2411.14774