Rainnet Eval

rainnet-eval

RainNet: A Large-Scale Imagery Dataset and Benchmark for Spatial Precipitation Downscaling — Chen et al. (2020) (arXiv:2012.09700, 2020)

What this evaluates

Evaluates deep learning models for spatial precipitation downscaling by measuring both static reconstruction accuracy and dynamic temporal evolution of rainfall patterns. It probes whether models can capture realistic meteorological properties like heavy rain coverage, cluster movement, and transition speeds.

Datasets

• RainNet — total 62400; splits: test (-1)

Metrics

• PEM (primary) — range: other
  • Precipitation Error Measure. Normalizes four reconstruction metrics (MPPE, HRRE, CPMSE, AMMD) by their Annual Mean Observations (AMO) to compute PBIAS, then averages them: PEM = 0.25 * Σ(|Metric_i| / |AMO_i|).
• PDEM — range: other
  • Precipitation Dynamics Error Measure. Normalizes two dynamic metrics (HRTS, CMD) by their AMO to compute PBIAS, then averages them: PDEM = 0.5 * Σ(|Metric_i| / |AMO_i|).

Input / output format

Input: Low-resolution spatial precipitation maps (rainfall intensity in mm/hour) across multiple time steps.

Output: High-resolution spatial precipitation maps (downscaled rainfall intensity in mm/hour) matching the ground truth resolution and time steps.

Scoring recipe

def compute_pem(pred, gt):
    pem = 0.0
    for m in ['MPPE', 'HRRE', 'CPMSE', 'AMMD']:
        amo = get_amo(m)  # Annual Mean Observation for metric m
        pem += 0.25 * abs(compute_metric(m, pred, gt)) / amo
    return pem

def compute_pdem(pred, gt):
    pdem = 0.0
    for m in ['HRTS', 'CMD']:
        amo = get_amo(m)
        pdem += 0.5 * abs(compute_metric(m, pred, gt)) / amo
    return pdem

Common pitfalls

• Using standard image super-resolution metrics (e.g., PSNR, SSIM) which fail to capture meteorological dynamics and temporal misalignment.
• Ignoring the need to normalize raw metrics by their Annual Mean Observations (AMO) before weighting, leading to scale-dominated scores.
• Evaluating single-frame outputs without considering temporal derivatives, which misses the dynamic precipitation properties the benchmark targets.

Evidence (verbatim from paper)

In order to further simplify the application of indices, we finally abstract them into two weighted and summed metrics: Precipitation Error Measure (PEM) and Precipitation Dynamics Error Measure (PDEM). We first align the dimensions of these two groups of metrics respectively. The first group of metrics (MPPE, HRRE, CPMSE, AMMD) is normalized, weighted , and summed to get the precipitation error measure (PEM). According to[[17]], all the metrics are transferred to Percent Bias (PBIAS) to be suitable for metrics weighting. The original definition of PBIAS is the bias divided by observation, as $PBIAS=|Q_{model}-Q_{obs}|/|Q_{obs}|$. Here we rewrite the original metrics to PBIAS by dividing the metrics with annual mean observations of the original variables (AMO), as $PBIAS_{i}^{PEM}=|Metrics_{i}^{PEM}|/|AMO_{i}^{PEM}|$, $Metrics_{i}^{PEM}={MPPE,HRRE,CPMSE,AMMD}$. ... The metrics then are ensembled to a single metric (PEM) with equal weight, as $PEM=\sum_{i}0.25\cdot PBIAS_{i}^{PEM}$. Following the same procedure, we then ensemble the second group of dynamic metrics (HRTS and CMD) to a single metrics $PDEM=\sum_{i}0.5\cdot PBIAS_{i}^{PDEM}$.

Citation

@misc{chen2020rainnet,
  title={RainNet: A Large-Scale Imagery Dataset and Benchmark for Spatial Precipitation Downscaling},
  author={Chen et al. (2020)},
  year={2020},
  note={arXiv:2012.09700}
}

• arXiv: 2012.09700