Ot Detection Eval

ot-detection-eval

Transparent Machine Learning: Training and Refining an Explainable Boosting Machine to Identify Overshooting Tops in Satellite Imagery — Mitchell et al. (2025) (arXiv:2507.03183, 2025)

What this evaluates

Evaluates a machine learning model's ability to detect overshooting tops (OTs) in satellite imagery at a 2 km pixel resolution. It measures how well the model predicts convection/OT presence using physics-informed features derived from visible and infrared channels.

Datasets

• GOES-16 ABI + MRMS Convection Labels — total ?; splits: test (2619)

Metrics

• hit, correct rejection, false alarm, miss counts (primary) — range: count
  • Pixel-level binary classification evaluated via confusion matrix counts: hit (model=1, label=1), correct rejection (model=0, label=0), false alarm (model=1, label=0), miss (model=0, label=1). Accuracy is derived as (hits + correct rejections) / total pixels.

Input / output format

Input: Per-pixel features extracted from GOES-16 ABI visible (Channel 2) and infrared (Channel 13) imagery, including brightness, infrared temperature, and Gray-Level Co-occurrence Matrix texture features.

Output: Binary classification per pixel indicating presence (1) or absence (0) of convection/overshooting top.

Scoring recipe

hits = 0; correct_rejections = 0; false_alarms = 0; misses = 0
for pred, label in zip(predictions, labels):
    if pred == 1 and label == 1: hits += 1
    elif pred == 0 and label == 0: correct_rejections += 1
    elif pred == 1 and label == 0: false_alarms += 1
    elif pred == 0 and label == 1: misses += 1
total = hits + correct_rejections + false_alarms + misses
accuracy = (hits + correct_rejections) / total

Common pitfalls

• Labels are derived from MRMS convection flags, not explicit OT annotations, so metrics actually measure convection prediction rather than OT detection.
• The 2 km prediction grid inherently causes spatial mismatch with labels, meaning some errors are unavoidable and not indicative of poor model performance.
• The VIS/IR sandwich product is used only for qualitative intuition and must not be used for quantitative validation.

Evidence (verbatim from paper)

When a prediction is made by the model, it is classified as one of the following: a “hit” if both the model and the convection labels indicated the presence of convection, a “correct rejection” if both the model and the convection labels indicated a lack of convection, a “false alarm” if the model indicated the presence of convection but the convection labels did not, and a “miss” if the model indicated a lack of convection but the convection labels did not. The finalized model achieved 30,755 hits, 10,481,845 correct rejections, 32,424 false alarms, and 182,400 misses.

Citation

@misc{mitchell2025transparent,
  title={Transparent Machine Learning: Training and Refining an Explainable Boosting Machine to Identify Overshooting Tops in Satellite Imagery},
  author={Mitchell et al. (2025)},
  year={2025},
  note={arXiv:2507.03183}
}

• arXiv: 2507.03183