Rainbench Eval

rainbench-eval

RainBench: Towards Global Precipitation Forecasting from Satellite Imagery — Christian Schroeder de Witt et al. (2020) (arXiv:2012.09670, 2020)

What this evaluates

Evaluates deep learning models' ability to forecast global precipitation at multiple lead times (1, 3, 5 days) and estimate same-timestep precipitation using multi-modal satellite and reanalysis data. It probes the model's capacity to handle extreme weather events, class imbalance, and spatial-temporal dependencies in meteorological forecasting.

Datasets

• RainBench — total ?; splits: train (-1), test (-1); repo https://github.com/frontierdevelopmentlab/pyrain

Metrics

• Latitude-weighted RMSE (primary) — range: other
  • Root Mean Squared Error weighted by latitude to account for grid cell area distortion near poles. Calculated as the square root of the mean squared difference between predicted and observed precipitation (mm), with weights proportional to the cosine of latitude.

Input / output format

Input: Multi-channel simulated satellite imagery (SimSat), ERA5 atmospheric state variables (e.g., specific humidity, temperature, geopotential height at various pressure levels), and static geographic features (land-sea mask, orography, soil type) arranged as spatial-temporal sequences.

Output: Global precipitation forecast maps in millimeters (mm) for specified lead times (1, 3, or 5 days) or same-timestep precipitation estimates.

Scoring recipe

def compute_latitude_weighted_rmse(pred, target, latitudes):
    weights = np.cos(np.radians(latitudes))
    weights = weights / np.sum(weights)
    mse = np.sum(weights * (pred - target) ** 2)
    return np.sqrt(mse)

Common pitfalls

• Ignoring latitude weighting leads to biased error estimates due to grid cell area distortion near poles.
• Deterministic models often produce blurry forecasts that underestimate extreme precipitation events due to class imbalance in training data.
• Using only satellite inputs without reanalysis data significantly degrades forecast skill compared to multi-modal fusion.

Evidence (verbatim from paper)

Table 2: Precipitation forecasts evaluated with Latitude-weighted RMSE (mm). All rows except where otherwise stated show models trained with data from 2016 onwards. Using the ERA5 precipitation as target, [2(a)] shows that training from SimSat alone gives the worst results across the data settings.

Citation

@misc{schroeder2020rainbench,
  title={RainBench: Towards Global Precipitation Forecasting from Satellite Imagery},
  author={Christian Schroeder de Witt et al. (2020)},
  year={2020},
  note={arXiv:2012.09670}
}

• arXiv: 2012.09670