Benchecg Eval

benchecg-eval

BenchECG and xECG: a benchmark and baseline for ECG foundation models — Lunelli et al. (2025) (arXiv:2509.10151, 2025)

What this evaluates

Evaluates ECG foundation models on diverse clinical tasks including classification, regression, detection, and survival analysis across multiple populations and signal lengths. It probes the model's ability to generalize across datasets, modalities (ECG vs PPG), and long-context temporal dependencies.

Datasets

• CODE-15% — total ?; splits: train (-1), val (-1), test (-1)
• Sleep-Apnea-ECG — total ?; splits: train (-1), val (-1), test (-1)
• MIT-BIH Arrhythmia — total ?; splits: train (-1), val (-1), test (-1)
• PTB-XL — total ?; splits: train (-1), val (-1), test (-1)
• CPSC2018 — total ?; splits: train (-1), val (-1), test (-1)
• MIMIC-IV-ECG — total ?; splits: train (-1), val (-1), test (-1)
• Exercise-ECG — total ?; splits: train (-1), test (-1)

Metrics

• BenchECG score (primary) — range: [0, 1]
  • The mean performance of a model across all tasks in the benchmark. Each task's metric is first normalized to the range [0, 1] before averaging.
• AUROC — range: [0, 1]
  • Area under the receiver operating characteristic curve, used for classification and segmentation tasks (except MIT-BIH).
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall. Used for the MIT-BIH classification task, detection tasks, and normalized for the composite score.
• SMAPE — range: percent
  • Symmetric mean absolute percentage error, used for regression tasks and normalized to [0, 1] for the composite score.
• C-index — range: [0, 1]
  • Concordance index for survival analysis, measuring the model's ability to correctly rank patients by risk.

Input / output format

Input: Raw ECG time-series signals (e.g., 10s fixed windows, 30-minute ambulatory recordings, or 12-lead clinical traces), sometimes paired with demographic or clinical metadata.

Output: Task-dependent predictions: class labels for classification, continuous values for regression, timestamp offsets for detection, or risk scores for survival analysis.

Scoring recipe

def compute_task_metric(preds, golds, task_type, task_name):
    if task_type in ['classification', 'segmentation']:
        return f1_score(golds, preds) if task_name == 'MIT-BIH' else auroc_score(golds, preds)
    elif task_type == 'regression':
        return 1.0 - smape(golds, preds)  # normalized to [0,1]
    elif task_type == 'detection':
        return f1_score(golds, preds, tolerance_ms=20)
    elif task_type == 'survival':
        return c_index(golds, preds)
    return 0.0

benchecg_score = mean([compute_task_metric(p, g, t, n) for n, t, p, g in tasks])

Common pitfalls

• Different datasets use different train/val/test splits; CODE-15% and Sleep-Apnea-ECG lack published validation splits and require custom random splits per run.
• Metric choice is task-dependent (AUROC, F1, SMAPE, C-index), so averaging requires normalization to [0,1] first.
• Linear probing and finetuning yield vastly different performance rankings, especially for long-context tasks.

Evidence (verbatim from paper)

To compare different foundation models, we propose the BenchECG score: the mean performance of a model across all tasks in the BenchECG benchmark. For each task we select a metric normalised to the range of 0 to 1. We use area under the receiver operator characteristic curve (AUROC) for classification and segmentation tasks, the symmetric mean absolute percentage error (SMAPE) for regression, F1 score for detection tasks, and the concordance index (C-index) for the survival analysis task

Citation

@misc{lunelli2025benchecg,
  title={BenchECG and xECG: a benchmark and baseline for ECG foundation models},
  author={Lunelli et al. (2025)},
  year={2025},
  note={arXiv:2509.10151}
}

• arXiv: 2509.10151