Flood Forecasting Eval

flood-forecasting-eval

A Spatiotemporal Radar-Based Precipitation Model for Water Level Prediction and Flood Forecasting — Dhankhar et al. (arXiv:2503.19943, 2025)

What this evaluates

Evaluates a model's ability to predict river water levels and forecast floods using spatiotemporal radar precipitation data. It probes the model's accuracy across multiple forecasting lead times (2h to 12h) and its robustness in capturing extreme hydrological events compared to baseline and deep learning models.

Datasets

• Goslar — total ?; splits: train (-1), test (-1)
• Göttingen — total ?; splits: train (-1), test (-1)

Metrics

• MSE — range: other
  • Mean Squared Error between predicted and observed water levels. Lower values indicate better accuracy.
• NSE (primary) — range: other
  • Nash-Sutcliffe Efficiency, measuring how well model predictions match the mean of observed data. Values closer to 1 indicate superior performance.
• IoA — range: [0, 1]
  • Index of Agreement assessing the degree of agreement between observed and predicted values, ranging from 0 (no agreement) to 1 (perfect agreement).
• Tok — range: other
  • Count of correct predictions where the absolute difference between predicted and observed water levels is within a ±10 cm tolerance.
• error_average — range: other
  • Absolute average deviation of predictions from observed values, reflecting the magnitude of prediction errors across events.

Input / output format

Input: High-resolution spatiotemporal radar precipitation data for a specific river location, processed through a (2+1)D CNN-LSTM architecture.

Output: Predicted water level (or change in water level, Δh_s) for forecasting horizons of 2, 3, 4, 8, and 12 hours.

Scoring recipe

def compute_metrics(pred, gold):
    mse = np.mean((pred - gold) ** 2)
    nse = 1 - np.sum((pred - gold)**2) / np.sum((gold - np.mean(gold))**2)
    ioa = 1 - np.sum((pred - gold)**2) / (np.sum(np.abs(pred - np.mean(gold)) + np.abs(gold - np.mean(gold)))**2)
    tok = np.sum(np.abs(pred - gold) <= 10)
    tover = np.sum(pred > gold + 10)
    tunder = np.sum(pred < gold - 10)
    error_avg = np.mean(np.abs(pred - gold))
    return mse, nse, ioa, tok, tover, tunder, error_avg

Common pitfalls

• Extrapolation to extreme flood events not present in the training dataset leads to significant accuracy drops, especially at longer horizons.
• Performance degrades noticeably for 8-12 hour forecasts compared to 2-4 hour forecasts; models should not be assumed equally accurate across all lead times.
• Event-focused metrics (Tok, Tover, Tunder) strictly require a ±10 cm tolerance threshold; ignoring this threshold invalidates the counts.
• Direct water level prediction (STRPM) performs poorly compared to residual-based prediction (STRPMr); evaluating absolute predictions without modeling changes yields misleadingly high errors.

Evidence (verbatim from paper)

The quantitative results in section 4.1 are evaluated using a range of performance metrics, including MSE, Bravais-Pearson (BP), Nash-Sutcliffe Efficiency (NSE), and Index of Agreement (IoA) to assess the accuracy and robustness of the proposed models.

Citation

@misc{dhankhar2025spatiotemporal,
  title={A Spatiotemporal Radar-Based Precipitation Model for Water Level Prediction and Flood Forecasting},
  author={Dhankhar et al.},
  year={2025},
  note={arXiv:2503.19943}
}

• arXiv: 2503.19943