Ptbx1 Ecg Statement Prediction Eval

ptbx1-ecg-statement-prediction-eval

Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL — Strodthoff et al. (2020) (arXiv:2004.13701, 2020)

What this evaluates

Evaluates deep learning models on 12-lead ECG time series for multi-label classification of diagnostic, rhythm, and form statements. It probes the ability of architectures to learn directly from raw signals versus traditional feature extraction, and assesses transfer learning and demographic attribute prediction capabilities.

Datasets

• PTB-XL — total ?; splits: train (-1), test (-1)

Metrics

• term-centric macro-averaged AUC (primary) — range: [0, 1]
  • Area under the ROC curve computed per label class and macro-averaged across all classes. Reflects discriminative performance independent of classification threshold.
• sample-centric Fmax-score — range: [0, 1]
  • Maximum F1-score achieved across all possible classification thresholds for each ECG sample, then averaged across samples.

Input / output format

Input: 12-lead ECG time series signals sampled at 100 Hz.

Output: Multi-label binary predictions for each ECG statement category (diagnostic, rhythm, form, etc.).

Scoring recipe

# Macro-averaged AUC across all label classes
auc_scores = [roc_auc_score(gold[:, c], pred[:, c]) for c in range(num_classes)]
macro_auc = np.mean(auc_scores)

# Sample-centric Fmax: max F1 across thresholds per sample, then averaged
fmax_scores = []
for i in range(num_samples):
    thresholds = np.unique(pred[i])
    best_f1 = max(f1_score(gold[i], (pred[i] >= t).astype(int), average='binary') for t in thresholds)
    fmax_scores.append(best_f1)
sample_fmax = np.mean(fmax_scores)

Common pitfalls

• Using random patient splits instead of the recommended patient-wise train-test splits, which causes data leakage.
• Optimizing models directly for Fmax or AUC instead of binary cross-entropy, which can lead to unstable training and overfitting to the evaluation metric.
• Ignoring the multi-label nature of the task by treating it as single-label classification or using accuracy instead of threshold-independent metrics.

Evidence (verbatim from paper)

Our proposed evaluation as described in Section II-C is applied the same way for each experiment, where we report the term-centric macro-averaged AUC and the sample-centric Fmax-score. In all six experiments, deep-learning-based methods show a high predictive performance. Interestingly, even though all models are optimized based on binary cross-entropy loss rather than on the target metrics directly, the ranking according to both sample-based and term-based metrics largely coincides across all algorithms, which is why we focus on macro AUC in the following.

Citation

@misc{strodthoff2020ptbxl,
  title={Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL},
  author={Strodthoff et al. (2020)},
  year={2020},
  note={arXiv:2004.13701}
}

• arXiv: 2004.13701