Scs Frequency Eval

scs-frequency-eval

Accessing Convective Hazards Frequency Shift with Climate Change using Physics-Informed Machine Learning — Mozikov et al. (2023) (arXiv:2310.03180, 2023)

What this evaluates

Evaluates machine learning models' ability to predict severe convective storm (SCS) frequency and occurrence in European Russia under climate change scenarios. It probes binary classification of SCS events against non-events, as well as regression accuracy on the normalized annual cycle of storm activity using physics-informed deep learning architectures.

Datasets

• CMIP5 RCP8.5 & Meteorological Observations — total 91375; splits: test (91375)

Metrics

• F1 (Class 1) — range: [0, 1]
  • Harmonic mean of precision and recall for the SCS class (Class 1).
• Recall (Class 1) — range: [0, 1]
  • True positive rate for the SCS class, calculated as TP / (TP + FN).
• RMSE — range: [0, ∞)
  • Root mean squared error between predicted and observed values.
• RMSEAC (primary) — range: [0, 1]
  • Root mean squared error of the normalized annual cycle of SCS activity, measuring how well the model captures seasonal trends.

Input / output format

Input: 3D geospatial time series of atmospheric features from multiple isobaric levels derived from CMIP5 climate projections.

Output: Binary classification label (Class 0: no SCS, Class 1: SCS) and continuous values representing the normalized annual cycle of SCS frequency.

Scoring recipe

def compute_metrics(pred_labels, gold_labels, pred_cycle, gold_cycle):
    tp = sum((p == 1) & (g == 1) for p, g in zip(pred_labels, gold_labels))
    fp = sum((p == 1) & (g == 0) for p, g in zip(pred_labels, gold_labels))
    fn = sum((p == 0) & (g == 1) for p, g in zip(pred_labels, gold_labels))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    rmseac = (sum((p - g) ** 2 for p, g in zip(pred_cycle, gold_cycle)) / len(gold_cycle)) ** 0.5
    return {'F1_Class1': f1, 'Recall_Class1': recall, 'RMSEAC': rmseac}

Common pitfalls

• Class 1 (SCS) is significantly underrepresented (23,360 vs 68,015 samples), so optimizing for overall accuracy or unweighted F1 will mask poor detection of severe events.
• The evaluation mixes real-world meteorological observations with CMIP5 climate model outputs, which simulate future trends rather than exact atmospheric states, making direct comparison with historical baselines challenging.
• RMSEAC measures error on the normalized annual cycle, not raw counts; failing to normalize predictions and targets before calculation will yield incorrect scores.

Evidence (verbatim from paper)

Table 2 summarizes the calculated metrics for all the considered models. ... This table shows that our two-part neural network has already surpassed the CatBoost baseline in terms of performance. ... Autoencoders & classifier (end-to-end) model performs best regarding the RMSEAC metric.

Citation

@misc{mozikov2023accessing,
  title={Accessing Convective Hazards Frequency Shift with Climate Change using Physics-Informed Machine Learning},
  author={Mozikov et al. (2023)},
  year={2023},
  note={arXiv:2310.03180}
}

• arXiv: 2310.03180