Tornet Eval

tornet-eval

A Benchmark Dataset for Tornado Detection and Prediction using Full-Resolution Polarimetric Weather Radar Data — Veillette et al. (2024) (arXiv:2401.16437, 2024)

What this evaluates

Evaluates machine learning models for detecting tornadoes using full-resolution polarimetric weather radar imagery. It probes the ability of classifiers to distinguish tornadic signatures from non-tornadic weather patterns across varying difficulty levels and threshold settings.

Datasets

• TorNet — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/mitll/tornet

Metrics

• AUC (ROC) (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, plotting True Positive Rate against False Positive Rate across all thresholds. Ranges from 0 to 1.
• CSI — range: [0, 1]
  • Critical Success Index = H / (H + M + F), where H=hits, M=misses, F=false alarms. Optimized over all thresholds to report a single value.
• Brier Score (BS) — range: [0, 1]
  • Mean squared difference between predicted probability p_i and observed binary outcome o_i: BS = (1/N) * sum(p_i - o_i). Lower is better.
• Accuracy (ACC) — range: [0, 1]
  • ACC = (H + C) / (H + C + M + F). Maximized over all thresholds for single-value reporting.

Input / output format

Input: Full-resolution polarimetric weather radar imagery (reflectivity factor, radial velocity, and other polarimetric variables) representing storm events.

Output: Continuous probability value 0 ≤ p ≤ 1 indicating likelihood of a tornado, which can be thresholded to a binary indicator.

Scoring recipe

def compute_metrics(pred_probs, labels, thresholds=np.linspace(0, 1, 100)):
    H, F, M, C = [], [], [], []
    for T in thresholds:
        preds = (pred_probs >= T).astype(int)
        H.append(np.sum((preds == 1) & (labels == 1)))
        F.append(np.sum((preds == 1) & (labels == 0)))
        M.append(np.sum((preds == 0) & (labels == 1)))
        C.append(np.sum((preds == 0) & (labels == 0)))
    H, F, M, C = np.array(H), np.array(F), np.array(M), np.array(C)
    tpr = H / (H + M + 1e-9)
    fpr = F / (F + C + 1e-9)
    sr = H / (H + F + 1e-9)
    csi = H / (H + M + F + 1e-9)
    acc = (H + C) / (H + C + M + F + 1e-9)
    bs = np.mean((pred_probs - labels)**2)
    auc = np.trapz(tpr, fpr)
    return {'AUC': auc, 'CSI': np.max(csi), 'ACC': np.max(acc), 'Brier_Score': bs}

Common pitfalls

• Accuracy is highly misleading due to severe class imbalance (a baseline always predicting 'No Tornado' achieves 93% accuracy).
• The dataset's label distribution is heavily biased towards tornadoes compared to real-world forecasting conditions, affecting probability calibration interpretation.
• Model performance varies significantly across different test set partitions (e.g., including vs excluding warning samples), so reporting a single aggregate metric without specifying the view is insufficient.

Evidence (verbatim from paper)

Given a threshold T, classifications over TorNet can be categorized and aggregated into number of true positives (or "hits") H(T), false positives (or "false alarms") F(T), false negatives (or "misses") M(T), and true negatives (or "correct rejections") C(T). These counts are used to compute the following metrics which are all dependent on the choice of threshold T (which we omit writing for conciseness): Accuracy: ACC = (H + C) / (H + C + M + F); true positive rate (i.e., probability of detection, recall): TPR = H / (H + M); false positive rate: FPR = F / (F + C); success rate (i.e., precision): SR = H / (H + F); and critical success index: CSI = H / (H + M + F).

Citation

@misc{veillette2024tornet,
  title={A Benchmark Dataset for Tornado Detection and Prediction using Full-Resolution Polarimetric Weather Radar Data},
  author={Veillette et al. (2024)},
  year={2024},
  note={arXiv:2401.16437}
}

• arXiv: 2401.16437