Synn Air Pollution Forecasting Eval

synn-air-pollution-forecasting-eval

Synergistic Neural Forecasting of Air Pollution with Stochastic Sampling — Abeysinghe et al. (2025) (arXiv:2510.23977, 2025)

What this evaluates

Evaluates a hybrid neural forecasting framework's ability to predict regional particulate matter (PM1, PM2.5, PM10) concentrations. It probes both average forecasting accuracy across spatial grids and the model's capacity to capture rare, high-impact pollution spikes and extreme events.

Datasets

• ERA5 & CAMS — total ?; splits: train (-1), test (-1)

Metrics

• Latitude-Weighted RMSE (primary) — range: other
  • Root Mean Square Error adjusted by latitude weights to correct for regional area disparities and avoid unfair advantages for local models during spatial averaging.
• RQE — range: other
  • Relative Quantile Error measuring deviation at high quantiles (90th–99.99th percentiles) to emphasize accuracy in capturing extremes. Negative values indicate underestimation.
• SEDI — range: [0, 1]
  • Symmetric Extremal Dependency Index capturing the skill in classifying extreme vs. non-extreme events, providing robustness across different percentile thresholds.

Input / output format

Input: Multi-variable meteorological (mslp, u10, v10, t2m) and PM (PM1, PM2.5, PM10) time-series data from ERA5/CAMS, spatially regridded to 0.4° or 5.26° resolution, with temporal context from previous timesteps.

Output: Predicted PM concentrations (PM1, PM2.5, PM10) in μg/m³ at specified spatial resolutions and lead times (1–6 days).

Scoring recipe

import numpy as np
# Latitude-Weighted RMSE
lat_weights = np.cos(np.deg2rad(lats))
lat_weights /= lat_weights.sum()
rmse = np.sqrt(np.mean(lat_weights * (pred - gold)**2))
# RQE
q = [0.90, 0.99, 0.9999]
rqe = np.mean((np.quantile(pred, q) - np.quantile(gold, q)) / np.quantile(gold, q))
# SEDI
thresh = np.quantile(gold, 0.90)
tp = np.sum((pred > thresh) & (gold > thresh))
fp = np.sum((pred > thresh) & (gold <= thresh))
fn = np.sum((pred <= thresh) & (gold > thresh))
tn = np.sum((pred <= thresh) & (gold <= thresh))
sed = 2 * (tp*tn - fp*fn) / ((tp+fp)*(fp+tn) + (tp+fn)*(fn+tn))

Common pitfalls

• Spatial averaging without latitude weighting unfairly advantages local models due to longitudinal span disparities.
• Comparing models at their native resolutions without harmonizing via regridding (e.g., ClimaX protocol) leads to invalid performance comparisons.
• Focusing solely on RMSE masks the diffusion module's true value, which is primarily reflected in tail-sensitive metrics like RQE and SEDI.

Evidence (verbatim from paper)

Metrics: We use three metrics to quantify performance, with a focus on both general accuracy and extreme event prediction (see Appendix C for formulations). Latitude-Weighted RMSE: Assesses forecast accuracy across spatial locations, correcting for regional area disparities. We adjust the longitudinal averaging to reflect reduced latitude span, avoiding unfair advantage for local models. RQE: Measures deviation at high quantiles (90th–99.99th percentiles), emphasizing the accuracy in capturing extremes. Negative values indicate underestimation. SEDI: Captures the skill in classifying extreme vs. non-extreme events, providing robustness across different percentile thresholds.

Citation

@misc{abeysinghe2025synn,
  title={Synergistic Neural Forecasting of Air Pollution with Stochastic Sampling},
  author={Abeysinghe et al. (2025)},
  year={2025},
  note={arXiv:2510.23977}
}

• arXiv: 2510.23977