Ecg Multitask Eval

ecg-multitask-eval

An Electrocardiogram Multi-task Benchmark with Comprehensive Evaluations and Insightful Findings — Xu et al. (2025) (arXiv:2512.08954, 2025)

What this evaluates

Evaluates the ability of foundation models (LLMs, time-series, and ECG-specific) and traditional deep learning models to perform regression and classification tasks on electrocardiogram (ECG) signals across zero-shot, few-shot, and fine-tuned settings.

Datasets

• ECG Multi-task Benchmark — total ?; splits: test (-1)

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error: the average of the absolute differences between predicted and true values. Lower is better.
• F1 Score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Higher is better.
• Accuracy (ACC) (primary) — range: [0, 1]
  • Proportion of correct predictions out of total predictions: sum(pred == true) / len(true). Higher is better.
• Win Rate — range: percent
  • Percentage of tasks or settings where a model achieves the top score compared to baselines. Higher is better.

Input / output format

Input: ECG signal segments (preprocessed or downsampled to match model input lengths), optionally accompanied by task-specific prompts or labels for supervised/fine-tuning settings.

Output: Predicted continuous values for regression tasks (RR interval, age), predicted class labels for classification tasks (gender, potassium abnormality, arrhythmia subtype), or saliency maps for interpretability.

Scoring recipe

def score(predictions, gold, task):
    if task in ['RR_interval', 'Age']:
        return mean(abs(predictions - gold))  # MAE
    elif task in ['Gender', 'Potassium']:
        tp = sum((predictions == 1) & (gold == 1))
        fp = sum((predictions == 1) & (gold == 0))
        fn = sum((predictions == 0) & (gold == 1))
        prec = tp / max(tp + fp, 1)
        rec = tp / max(tp + fn, 1)
        return 2 * prec * rec / (prec + rec)  # F1
    elif task == 'Arrhythmia':
        return sum(predictions == gold) / len(gold)  # ACC
    elif task == 'Benchmark':
        wins = sum(1 for m in predictions if m == max(predictions))
        return wins / len(predictions)  # Win Rate

Common pitfalls

• LLMs struggle with raw time-series ECG data without extensive prompt engineering or feature extraction, often performing worse than traditional time-series models.
• Zero-shot and few-shot settings frequently yield suboptimal results for foundation models due to domain shifts between pretraining corpora and clinical ECG distributions.
• Fine-tuning requires sufficient labeled samples; limited tuning can hinder effective adaptation to ECG-specific tasks.

Evidence (verbatim from paper)

Table 2: Benchmarking experimental results. Highlighted are the top first, second, and third results. (RR Interval Estimation, Age Estimation, Gender Classification, Potassium Abnormality Prediction, Arrhythmia Detection, and zero-shot, few-shot, fine-tune are denoted as RR., Age, Gen., Ka, AD, and zs, fs, ft respectively.) Regre. (MAE)↓ | Binary Class (F1 Score)↑ | 15 Class (ACC)↑ | Benchmark (Win Rate)↑

Citation

@misc{xu2025ecgmultitask,
  title={An Electrocardiogram Multi-task Benchmark with Comprehensive Evaluations and Insightful Findings},
  author={Xu et al. (2025)},
  year={2025},
  note={arXiv:2512.08954}
}

• arXiv: 2512.08954