Sepnet Eval

sepnet-eval

Solar Energetic Particle Forecasting with Multi-Task Deep Learning: SEPNET — Yu et al. (2025) (arXiv:2512.12786, 2025)

What this evaluates

Evaluates deep learning models for forecasting solar energetic particle (SEP) events using solar magnetic field parameters and historical eruptive features. It probes the model's ability to classify general and operational SEP occurrences under different feature sets and temporal conditions.

Datasets

• SEPVAL — total ?; splits: test (-1); repo https://github.com/yuyian/SEP-Prediction.git
• CLEAR — total ?; splits: train (-1), test (-1); repo https://github.com/yuyian/SEP-Prediction.git

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall. Calculated as 2 * TP / (2 * TP + FP + FN).
• TSS — range: [-1, 1]
  • True Skill Score, also known as Hanssen-Kuipers discriminant. Measures the proportion of correctly detected events minus the proportion of false alarms.
• HSS — range: [-1, 1]
  • Heidke Skill Score, measures the accuracy of the forecast relative to random chance. Ranges from -1 to 1, with 1 indicating perfect skill.
• POD — range: [0, 1]
  • Probability of Detection, equivalent to recall or sensitivity. Calculated as TP / (TP + FN).
• FAR — range: [0, 1]
  • False Alarm Rate, measures the fraction of predicted events that did not occur. Calculated as FP / (FP + TN).
• ACC — range: [0, 1]
  • Overall accuracy, the fraction of correct predictions out of total samples.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve, measuring the model's ability to discriminate between classes across all thresholds.

Input / output format

Input: Temporal sequences of SHARP magnetic field parameters, optionally combined with historical flare and CME features.

Output: Binary classification (SEP event vs. non-SEP) or continuous probability score. A decision threshold is optimized to maximize HSS for operational forecasting.

Scoring recipe

def compute_metrics(y_true, y_pred, threshold=0.5):
    y_bin = (y_pred >= threshold).astype(int)
    tp = np.sum((y_bin == 1) & (y_true == 1))
    fn = np.sum((y_bin == 0) & (y_true == 1))
    fp = np.sum((y_bin == 1) & (y_true == 0))
    tn = np.sum((y_bin == 0) & (y_true == 0))
    acc = (tp + tn) / (tp + tn + fp + fn)
    f1 = 2 * tp / (2 * tp + fp + fn)
    pod = tp / (tp + fn)
    far = fp / (fp + tn)
    tss = pod + (tn / (tn + fp)) - 1
    hss = (tp * tn - fp * fn) / ((tp + fp) * (fp + tn) + (tp + fn) * (fn + tn))
    return {'ACC': acc, 'F1': f1, 'POD': pod, 'FAR': far, 'TSS': tss, 'HSS': hss}

Common pitfalls

• Class imbalance in solar event data leads to elevated false alarm rates (high FAR) even when detection rates (POD) are high.
• Real-time SHARP parameters differ from definitive HARP data used in training, causing systematic underestimation of flare counts and prediction mismatches.
• Threshold optimization for operational SEP events is performed on the training/validation split, which may not generalize perfectly to out-of-time real-world data.

Evidence (verbatim from paper)

For a rigorous comparison with the state-of-the-art pre-eruptive models (denoted as SoA) on SEPVAL [Whitman2026], we performed 50 independent runs for each configuration to derive the median and $75$th percentile (target quantile) metrics. Our models incorporating SHARP parameters generally match or outperform the SoA benchmarks in terms of standard evaluation metrics such as ACC, AUC, F1, POD, TSS, and HSS.

Citation

@misc{yu2025sepnet,
  title={Solar Energetic Particle Forecasting with Multi-Task Deep Learning: SEPNET},
  author={Yu et al. (2025)},
  year={2025},
  note={arXiv:2512.12786}
}

• arXiv: 2512.12786