Ef4inca Nowcast Eval

ef4inca-nowcast-eval

Integrated nowcasting of convective precipitation with Transformer-based models using multi-source data — Küçük et al. (2024) (arXiv:2409.10367, 2024)

What this evaluates

Evaluates the capability of spatiotemporal Transformer models to nowcast convective precipitation up to 90 minutes ahead using multi-source meteorological data. It probes the model's ability to fuse satellite infrared, radar, and NWP inputs to accurately predict the initiation, location, and intensity of rapidly evolving convective cells.

Datasets

• Austria convective precipitation dataset — total ?; splits: test (-1); repo https://github.com/caglarkucuk/earthformer-multisource-to-inca

Metrics

• Critical Success Index (CSI) (primary) — range: [0, 1]
  • Measures the ratio of correctly predicted precipitation events to the total number of predicted and observed events. It balances hits and false alarms.
• Probability of Detection (POD) — range: [0, 1]
  • Also known as hit rate. Computes the fraction of observed precipitation events that were correctly predicted.
• Fractional Skill Score (FSS) — range: [0, 1]
  • Evaluates prediction skill by comparing the fraction of grid cells exceeding a threshold in predictions versus observations, smoothed over a specified spatial scale (e.g., 5 km or 30 km).

Input / output format

Input: Multi-channel spatiotemporal tensor containing SEVIRI satellite infrared channels (CH5-CH9), ground-based radar analysis (INCA_a), numerical weather prediction fields (INCA, CAPE), and lightning data, all at 1 km × 5 min resolution.

Output: Predicted precipitation rate field (mm/h) for each lead time step up to 90 minutes ahead.

Scoring recipe

def compute_csi(pred, obs, threshold):
    pred_bin = (pred >= threshold).astype(int)
    obs_bin = (obs >= threshold).astype(int)
    hits = np.sum((pred_bin == 1) & (obs_bin == 1))
    fa = np.sum((pred_bin == 1) & (obs_bin == 0))
    misses = np.sum((pred_bin == 0) & (obs_bin == 1))
    return hits / (hits + fa + misses) if (hits + fa + misses) > 0 else 0.0

def compute_fss(pred, obs, threshold, kernel_km):
    pred_smooth = smooth_binary(pred, kernel_km)
    obs_smooth = smooth_binary(obs, kernel_km)
    return 1.0 - np.sum((pred_smooth - obs_smooth)**2) / (np.sum(pred_smooth**2) + np.sum(obs_smooth**2))

Common pitfalls

• Grid-cell level metrics like CSI and POD heavily penalize minor spatial displacements, making location accuracy appear worse than it is.
• FSS results are highly sensitive to the chosen spatial scale/kernel size (e.g., 5 km vs 30 km), which can reverse model rankings.
• Performance varies significantly across precipitation thresholds; models may excel at low thresholds but fail at high thresholds, or vice versa.

Evidence (verbatim from paper)

The probability of detection [POD; Hogan and Mason, 2011], also known as hit rate, scores for various thresholds are summarised in Figure 3a. While the variance-preserving models received higher POD scores in thresholds smaller than 1mm / h , EF4INCA scored higher in larger thresholds. Critical success index [CSI; Hogan and Mason, 2011], also known as the threat score, was also computed in order to consider event detection and false alarms simultaneously. Unlike POD, EF4INCA received higher CSI scores than the other two models across all thresholds (Figure 3b). This suggests superior performance of EF4INCA in predicting location of the precipitation fields without delivering too many false alarms, particularly in low precipitation values. CSI and POD are computed at grid cell level, which heavily penalises location inaccuracies. Therefore, we also computed the fractional skill score [FSS; Roberts and Lean, 2008], which accounts for neighbouring grid cells when computing prediction skill.

Citation

@misc{kucuk2024ef4inca,
  title={Integrated nowcasting of convective precipitation with Transformer-based models using multi-source data},
  author={Küçük et al. (2024)},
  year={2024},
  note={arXiv:2409.10367}
}

• arXiv: 2409.10367