Nfip Flood Loss Eval

nfip-flood-loss-eval

Learning Inter-Annual Flood Loss Risk Models From Historical Flood Insurance Claims and Extreme Rainfall Data — Salas et al. (2022) (arXiv:2212.08660, 2022)

What this evaluates

Evaluates machine learning regressors on predicting inter-annual flood economic loss using historical insurance claims and meteorological data. It probes both pointwise prediction accuracy and the fidelity of the predicted loss distribution compared to ground truth, emphasizing temporal generalization over random splits.

Datasets

• NFIP (National Flood Insurance Program) — total ?; splits: train (-1), test (-1)

Metrics

• R² (primary) — range: [0, 1]
  • Coefficient of determination. The paper defines a distributional variant: R² = 1 - S_r/S_v, where S_r is the integrated squared difference between reference and prediction PDFs, and S_v is the integrated squared deviation of the reference PDF from its mean. Standard regression R² is also used for pointwise evaluation.
• RMSE — range: other
  • Root-mean-squared error measuring pointwise similarity: RMSE = sqrt(1/n * Σ(y_pred - y_true)²). Often normalized by the standard deviation of the response (RMSE/σ) to enable cross-county comparison.
• K-S statistic — range: [0, 1]
  • Kolmogorov-Smirnov test statistic measuring distributional distance: D_n = max_x |P_r(x) - Q_p(x)|, where P_r and Q_p are the cumulative distribution functions of reference and predicted losses.
• KL divergence — range: other
  • Kullback-Leibler divergence measuring asymmetry between reference and prediction PDFs: D(p_r || q_p) = ∫ p_r(x) log(p_r(x)/q_p(x)) dx.

Input / output format

Input: Historical flood claim records (amounts, dates, county locations) and aggregated daily precipitation data from Daymet. Precipitation is aggregated within ±0.05° of claim location using sum or max over 2, 4, or 6 days prior to the event. Continuous predictors are normalized to zero mean and unit variance.

Output: Predicted flood loss amount (continuous scalar) or parameters of a fitted Burr distribution for distributional modeling.

Scoring recipe

def evaluate(y_true, y_pred, p_true, p_pred):
    # Pointwise metrics
    rmse = np.sqrt(np.mean((y_pred - y_true)**2))
    r2_pointwise = 1 - np.sum((y_pred - y_true)**2) / np.sum((y_true - np.mean(y_true))**2)
    
    # Distributional metrics
    ks_stat = np.max(np.abs(np.sort(p_pred) - np.sort(p_true)))
    kl_div = np.sum(p_true * np.log(p_true / p_pred))
    
    # Distributional R² (Eq 19)
    s_r = np.sum((p_true - p_pred)**2)
    s_v = np.sum((p_true - np.mean(p_true))**2)
    r2_dist = 1 - s_r / s_v
    
    return {'RMSE': rmse, 'R2_pointwise': r2_pointwise, 'R2_distributional': r2_dist, 'KS': ks_stat, 'KL': kl_div}

Common pitfalls

• Using random train/test splits instead of the specified time-series shifting/expanding protocol, which violates the temporal causality required for flood loss prediction.
• Focusing exclusively on pointwise metrics (RMSE/R²) while ignoring distributional fidelity (K-S/KL), which is critical for risk modeling and insurance pricing.
• Failing to normalize continuous predictors (like rainfall) to zero mean and unit variance before training, as explicitly required by the protocol.

Evidence (verbatim from paper)

To measure the pointwise similarity between the predictions for the response variable and the corresponding reference values, we employ indicators including the root-mean-squared error (RMSE), the RMSE divided by the standard deviation σ of the response value, and the coefficient of determination... we assess the difference between the reference and prediction distributions using the Kullback-Leibler (KL) divergence... Another insightful measure of performance is the determination coefficient R^2... After bias correction, when there is a parametric representation of the reference p_r(x) and prediction q_p(x) distributions, we express R^2 as R^2 = 1 - S_r/S_v = 1 - (∫(p_r(x) - q_p(x))^2 dx)/(∫(p_r(x) - μ_r)^2 dx)

Citation

@misc{salas2022floodloss,
  title={Learning Inter-Annual Flood Loss Risk Models From Historical Flood Insurance Claims and Extreme Rainfall Data},
  author={Salas et al. (2022)},
  year={2022},
  note={arXiv:2212.08660}
}

• arXiv: 2212.08660