Ecg Fm Benchmark Eval

ecg-fm-benchmark-eval

Benchmarking ECG Foundational Models: A Reality Check Across Clinical Tasks — Al-Masud et al. (2025) (arXiv:2509.25095, 2025)

What this evaluates

Evaluates the clinical utility and label efficiency of ECG foundation models across diverse tasks including adult/pediatric ECG interpretation, cardiac structure prediction, clinical outcome forecasting, and patient characteristic regression. It probes cross-domain generalization, fine-tuning adaptability, and the quality of frozen/linear representations compared to strong supervised baselines.

Datasets

• PTB-XL — total ?; splits: test (-1)
• EchoNext — total ?; splits: test (-1)
• MIMIC-IV (ECG) — total ?; splits: test (-1)
• CPSC2018 — total ?; splits: test (-1)
• PTB — total ?; splits: test (-1)
• Ningbo — total ?; splits: test (-1)
• Georgia — total ?; splits: test (-1)
• Chapman — total ?; splits: test (-1)
• SPH — total ?; splits: test (-1)
• CODE-15% — total ?; splits: test (-1)
• ZZU pECG — total ?; splits: test (-1)

Metrics

• macro-AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve, averaged across all classes (macro-averaged). Higher values indicate better classification performance.
• average z-normalized MAE (primary) — range: [0, inf)
  • Mean Absolute Error computed on z-normalized target and prediction vectors: MAE(z(y_true), z(y_pred)). Normalization mitigates scale differences across regression targets.
• label efficiency ratio — range: [0, 1]
  • Ratio r = N*/N, where N* is the training set size required for a pretrained model to match the performance of a supervised baseline trained on N samples. Derived from parametric scaling curve fits.

Input / output format

Input: ECG time-series signals (typically 12-lead), optionally with patient metadata for regression tasks.

Output: Class labels for interpretation/outcome tasks; continuous values for biometric/clinical regression tasks.

Scoring recipe

def macro_auroc(y_true, y_pred):
    scores = [roc_auc_score(y_true[:, i], y_pred[:, i]) for i in range(y_true.shape[1])]
    return np.mean(scores)

def z_norm_mae(y_true, y_pred):
    z_true = (y_true - np.mean(y_true)) / np.std(y_true)
    z_pred = (y_pred - np.mean(y_pred)) / np.std(y_pred)
    return np.mean(np.abs(z_true - z_pred))

Common pitfalls

• Comparing frozen/linear evaluation directly with finetuning without acknowledging that they measure different aspects of representational quality.
• Using raw MAE instead of z-normalized MAE for regression tasks, which unfairly penalizes models on targets with high variance or different scales.
• Ignoring stratified subsampling protocols when evaluating label efficiency, leading to biased scaling curves.

Evidence (verbatim from paper)

Table 2: Comparison of macro-AUROC (classification) and average z-normalized MAE (regression) under finetuning with linear prediction head. ... We use the parametric form of the fits to work out a label efficiency ratio r=N*/N, i.e., the fraction of samples N* required for the pretrained model to reach the same performance as the supervised baseline for given N.

Citation

@misc{almasud2025ecgfmbenchmarking,
  title={Benchmarking ECG Foundational Models: A Reality Check Across Clinical Tasks},
  author={Al-Masud et al. (2025)},
  year={2025},
  note={arXiv:2509.25095}
}

• arXiv: 2509.25095