Climdetect Eval

climdetect-eval

ClimDetect: A Benchmark Dataset for Climate Change Detection and Attribution — Sungduk Yu et al. (2024) (arXiv:2408.15993, 2024)

What this evaluates

This benchmark evaluates machine learning models' ability to detect and attribute human-induced climate change signals from daily spatial climate fields. It specifically probes how well models can predict Annual Global Mean Temperature (AGMT) using surface temperature, humidity, and precipitation data, and assesses their sensitivity in identifying the year when climate change signals robustly emerge from natural variability.

Datasets

• ClimDetect — total 1170000; splits: train (-1), val (-1), test (-1)

Metrics

• RMSE (primary) — range: other
  • Root Mean Square Error, calculated as the square root of the mean squared difference between predicted and actual Annual Global Mean Temperature (AGMT). Serves as a reliable proxy for detection sensitivity in climate science.
• Year of Emergence (YoE) — range: other
  • The first calendar year where the emergence fraction (EF) exceeds 97.5%. EF is the fraction of days in a year where the predicted AGMT exceeds the 97.5th percentile of natural variability estimated from historical CMIP6 simulations.

Input / output format

Input: Daily spatial climate field snapshots (grids) of surface temperature (tas), surface humidity (huss), and total precipitation rate (pr). Inputs may optionally have their spatial mean removed (mean-removed/mr setup).

Output: A single scalar value representing the predicted Annual Global Mean Temperature (AGMT) in degrees Celsius (°C).

Scoring recipe

def compute_rmse(predictions, gold):
    import numpy as np
    return np.sqrt(np.mean((np.array(predictions) - np.array(gold)) ** 2))

Common pitfalls

• Hyperparameters were tuned exclusively on the ViT-b/16 model with the 'tas-huss-pr' configuration, which may artificially favor that specific setup over others.
• The 'mean-removed' (mr) experiments strip the global mean from inputs, shifting the task from absolute temperature prediction to detecting spatial anomaly patterns, which changes model behavior significantly.
• Models relying solely on precipitation ('pr_only') consistently fail across all architectures due to the sparse and indirect relationship between precipitation and temperature/humidity variables.

Evidence (verbatim from paper)

In assessing the performance of our baseline models on the withheld test split, we used RMSE as the primary evaluation metric—one of the most widely used metrics in climate science. RMSE has proven to be a reliable proxy for detection sensitivity.

Citation

@misc{yu2024climdetect,
  title={ClimDetect: A Benchmark Dataset for Climate Change Detection and Attribution},
  author={Sungduk Yu et al. (2024)},
  year={2024},
  note={arXiv:2408.15993}
}

• arXiv: 2408.15993