Ptbxl Af Detection Eval

ptbxl-af-detection-eval

Sampling Matters: The Effect of ECG Frequency on Deep Learning-Based Atrial Fibrillation Detection — Arjan Mahmuod et al. (2026) (arXiv:2604.16437, 2026)

What this evaluates

This benchmark evaluates how ECG sampling frequency impacts deep learning models for binary atrial fibrillation detection. It probes a model's discrimination capability and the reliability of its predicted probabilities across different temporal resolutions, while enforcing strict patient-level separation to prevent data leakage.

Datasets

• PTB-XL — total 11005; splits: train_val (7053), test (3023)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve. Measures the model's ability to rank positive instances higher than negative ones across all classification thresholds.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified instances (true positives + true negatives) out of the total number of instances.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall at a 0.5 decision threshold.
• Brier score — range: [0, 1]
  • Mean squared difference between predicted probabilities and actual binary outcomes. Lower values indicate better probabilistic accuracy.
• ECE — range: [0, 1]
  • Expected Calibration Error. Computed by partitioning predicted probabilities into equally spaced confidence bins and calculating the weighted absolute difference between the average predicted confidence and the empirical accuracy within each bin.

Input / output format

Input: 12-lead, 10-second resting ECG recordings resampled to 62, 100, 250, or 500 Hz using FFT-based interpolation, followed by per-lead Z-score standardization. Input sequence length scales linearly with sampling frequency (T = 10 × f_s).

Output: Softmax class probabilities over two classes (NORM, AFIB). The probability for the positive class (AFIB) is extracted as softmax(z)[:,1] from the model logits.

Scoring recipe

def compute_metrics(probs, labels):
    pos_probs = probs[:, 1]
    preds = (pos_probs >= 0.5).astype(int)
    acc = np.mean(preds == labels)
    f1 = f1_score(labels, preds)
    auroc = roc_auc_score(labels, pos_probs)
    brier = np.mean((pos_probs - labels)**2)
    bin_edges = np.linspace(0, 1, 11)
    ece = 0.0
    for i in range(10):
        mask = (pos_probs >= bin_edges[i]) & (pos_probs < bin_edges[i+1])
        if mask.sum() > 0:
            bin_acc = labels[mask].mean()
            bin_conf = pos_probs[mask].mean()
            ece += mask.sum() / len(labels) * abs(bin_acc - bin_conf)
    return {'AUROC': auroc, 'Accuracy': acc, 'F1': f1, 'Brier': brier, 'ECE': ece}

Common pitfalls

• Failing to enforce patient-level splitting, which causes data leakage and artificially inflates performance metrics.
• Applying a decision threshold to probabilities when computing calibration metrics like ECE, contradicting the protocol's requirement to use raw softmax outputs directly.
• Ignoring the natural class imbalance in the test set, as the benchmark explicitly evaluates on the real clinical prevalence rather than a balanced subset.

Evidence (verbatim from paper)

Model discrimination was assessed using the Area Under the Receiver Operating Characteristic curve (AUROC), classification accuracy, and F1-score. Calibration and Robustness: Beyond discrimination, we evaluated the reliability of predicted probabilities using calibration curves, the Brier score, and Expected Calibration Error (ECE). Validation logits were converted to class probabilities using softmax, and the resulting probabilities were stored for each cross-validation fold. ECE was then computed post-hoc by partitioning predicted probabilities into equally spaced confidence bins and measuring the weighted absolute difference between average confidence and empirical accuracy within each bin.

Citation

@misc{mahmuod2026sampling,
  title={Sampling Matters: The Effect of ECG Frequency on Deep Learning-Based Atrial Fibrillation Detection},
  author={Arjan Mahmuod et al. (2026)},
  year={2026},
  note={arXiv:2604.16437}
}

• arXiv: 2604.16437