Ecg Multi Label Eval

ecg-multi-label-eval

ECG-Soup: Harnessing Multi-Layer Synergy for ECG Foundation Models — Phu X. Nguyen et al. (2025) (arXiv:2509.00102, 2025)

What this evaluates

This evaluation probes the ability of ECG foundation models to learn robust, generalizable representations from unsupervised pretraining and transfer them to downstream multi-label classification tasks. It specifically tests generalization across different clinical datasets and sampling rates by measuring performance on arrhythmia conditions and rhythm classifications.

Datasets

• PTB-XL — total 21837; splits: train (-1), val (-1), test (-1)
• Chapman — total 10646; splits: train (-1), val (-1), test (-1)

Metrics

• macro AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve computed per class and averaged across all classes (macro average).
• macro F1-score — range: [0, 1]
  • F1-score computed per class and averaged across all classes (macro average).
• sample accuracy — range: [0, 1]
  • Accuracy computed per sample (instance) and then averaged across all samples.

Input / output format

Input: 12-lead ECG signals normalized to a fixed 100 Hz sampling rate and 10-second duration (truncated or zero-padded).

Output: Binary probability scores for each of the 71 PTB-XL conditions, 67 Chapman conditions, 12 PTB-XL rhythms, or 11 Chapman rhythms.

Scoring recipe

import numpy as np
from sklearn.metrics import average_precision_score, f1_score, accuracy_score

def evaluate(y_true, y_pred):
    # y_true: (N, C) binary ground truth
    # y_pred: (N, C) predicted probabilities
    macro_auc = average_precision_score(y_true, y_pred, average='macro')
    macro_f1 = f1_score(y_true, (y_pred > 0.5).astype(int), average='macro')
    sample_acc = accuracy_score(y_true.flatten(), (y_pred > 0.5).astype(int).flatten())
    return {'macro_auc': macro_auc, 'macro_f1': macro_f1, 'sample_acc': sample_acc}

Common pitfalls

• Data leakage between pretraining and evaluation: CinC2020 contains PTB-XL recordings. The OOD scenario explicitly removes PTB-XL from pretraining, but in-distribution evaluation retains it, requiring careful dataset curation.
• Signal preprocessing mismatch: All inputs must be resampled to 100 Hz and padded/truncated to exactly 10 seconds. Deviations break the fixed-length ViT patching mechanism.
• Multi-label metric aggregation: Metrics are reported as both macro (per-label averaged) and sample (per-sample averaged). Confusing these aggregation methods leads to incorrect benchmarking.

Evidence (verbatim from paper)

Model performance was assessed using both macro- and sample-level metrics, including macro/sample AUC, instance/sample accuracy, and macro/sample F1-score, providing a comprehensive evaluation of the models in multi-label classification settings.

Citation

@misc{nguyen2025ecgsoup,
  title={ECG-Soup: Harnessing Multi-Layer Synergy for ECG Foundation Models},
  author={Phu X. Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2509.00102}
}

• arXiv: 2509.00102