Physionet Cinc Ecg Eval

physionet-cinc-ecg-eval

One Dimensional CNN ECG Mamba for Multilabel Abnormality Classification in 12 Lead ECG — Jiang et al. (2025) (arXiv:2510.13046, 2025)

What this evaluates

Evaluates a model's ability to perform multilabel classification of cardiac abnormalities from 12-lead ECG recordings. It specifically probes robustness to severe class imbalance and the capacity to handle diverse lead configurations and sampling rates in long-sequence cardiac signals.

Datasets

• PhysioNet/CinC Challenge 2021 — total 43101; splits: train (-1), test (-1)

Metrics

• Macro AUPRC (primary) — range: [0, 1]
  • Macro-averaged Area Under the Precision-Recall Curve. Computed by calculating the AUPRC for each of the 26 classes independently and then averaging them, giving equal weight to minority and majority classes.
• Macro AUROC — range: [0, 1]
  • Macro-averaged Area Under the Receiver Operating Characteristic Curve. Measures the classifier’s ability to discriminate between positive and negative instances for each class, then averages the scores across all 26 categories.

Input / output format

Input: Raw 12-lead ECG time-series recordings (sampled at 500 Hz, 1000 Hz, or 257 Hz), with support for diverse lead configurations (6, 4, 3, or 2-lead formats).

Output: Multilabel probability scores or binary predictions across 26 classes (25 cardiac abnormalities + sinus rhythm).

Scoring recipe

# preds: (N, 26) float probabilities
# gold: (N, 26) binary labels
auprcs = [auprc(gold[:, c], preds[:, c]) for c in range(26)]
aurocs = [auroc(gold[:, c], preds[:, c]) for c in range(26)]
macro_auprc = sum(auprcs) / 26
macro_auroc = sum(aurocs) / 26

Common pitfalls

• Treating the task as single-label classification instead of multilabel, which misrepresents clinical reality where patients often have multiple comorbidities.
• Using micro-averaging or accuracy instead of macro-averaged AUPRC/AUROC, which would bias results toward majority classes and ignore the severe class imbalance.
• Failing to average the 5-fold cross-validation results, leading to high variance and unreliable performance estimates.
• Confusing the 2020 and 2021 dataset splits, as the 2021 dataset fully contains the 2020 dataset but has different class counts (26 vs 27) and sizes.

Evidence (verbatim from paper)

To assess the performance of the ECG classification model under the challenges of class imbalance and multi-label prediction, two evaluation metrics were employed: Macro Area Under the Precision–Recall Curve (AUPRC): This metric evaluates the predictive capacity of the model in all categories, with a particular emphasis on minority classes. By averaging precision–recall performance over all labels, Macro AUPRC is well-suited for imbalanced datasets, as it effectively reflects the trade-off between precision and recall under underrepresented conditions. Macro Area Under the Receiver Operating Characteristic Curve (AUROC): AUROC measures the classifier’s ability to discriminate between positive and negative instances while treating each class with equal importance. The macro-averaged formulation ensures that performance is not biased toward the majority classes, thus providing a balanced assessment of the discriminative power of the model.

Citation

@misc{jiang2025onedimensional,
  title={One Dimensional CNN ECG Mamba for Multilabel Abnormality Classification in 12 Lead ECG},
  author={Jiang et al. (2025)},
  year={2025},
  note={arXiv:2510.13046}
}

• arXiv: 2510.13046