Camel Ecg Eval

camel-ecg-eval

CAMEL: An ECG Language Model for Forecasting Cardiac Events — Velingker et al. (2026) (arXiv:2602.15677, 2026)

What this evaluates

Evaluates a multimodal language model's ability to process long-duration ECG signals for clinical forecasting, diagnostic classification, report generation, and statistical grounding. It probes temporal reasoning, multi-lead interpretation, and instruction-following in medical AI settings.

Datasets

• Icentia11k — total ?; splits: test (-1)
• PTB-XL — total ?; splits: test (-1)
• CSN — total ?; splits: test (-1)
• CODE-15% — total ?; splits: test (-1)
• CPSC-2018 — total ?; splits: test (-1)
• HEEDB — total ?; splits: test (-1)
• Penn — total ?; splits: test (-1)
• MIMIC-IV-ECG — total ?; splits: test (-1)
• ECGBench / ECG-QA — total ?; splits: test (-1)
• ECG Grounding Benchmark — total 2000; splits: test (2000)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall. Reported as macro-average across classes or forecasting horizons.
• AUROC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive and false positive rates for linear probing.
• RMSE — range: other
  • Root Mean Squared Error between predicted and ground-truth ECG statistics (e.g., RR interval, HR, QRS duration).
• Accuracy — range: [0, 1]
  • Fraction of correctly predicted multiple-choice answers or class labels.
• BLEU-1/4, METEOR, Rouge, BERT-F1 — range: [0, 1]
  • Standard NLP sequence matching and embedding-based metrics for evaluating generated clinical reports.
• LLM-as-a-judge score — range: other
  • Scored by GPT-5 on diagnostic accuracy, analysis completeness, and relevance based on predefined criteria (e.g., 30-point scale).

Input / output format

Input: Raw ECG waveforms (1-lead or multi-lead, sampled at 256 Hz) tokenized into 1-second segments, interleaved with text prompts, questions, or instruction templates.

Output: Predicted class labels or probabilities, generated clinical reports, multiple-choice answers, or numerical statistical values (e.g., RR interval, HR).

Scoring recipe

def compute_metrics(preds, gold):
    f1 = f1_score(gold, preds, average='macro')
    auroc = roc_auc_score(gold, preds_proba)
    rmse = np.sqrt(np.mean((gold - preds)**2))
    accuracy = np.mean(preds == gold)
    nlp = compute_nlp_metrics(preds, gold)
    llm = llm_as_judge(preds, gold, criteria=['accuracy', 'completeness', 'relevance'])
    return {'F1': f1, 'AUROC': auroc, 'RMSE': rmse, 'Accuracy': accuracy, 'NLP': nlp, 'LLM_Judge': llm}

Common pitfalls

• Forecasting F1 drops as the prediction horizon increases, but improves with longer input windows (up to 600s).
• LLM-based baselines struggle on datasets with large label spaces (e.g., HEEDB, CSN) compared to non-LLM architectures.
• Linear probing evaluation freezes the model and trains a linear classifier on only 1% of the training data.
• LLM-as-a-judge scoring relies on GPT-5 to evaluate diagnostic accuracy, completeness, and relevance, which may introduce model-specific biases.

Evidence (verbatim from paper)

We report F1 score for zero-shot evaluation and AUROC for linear probing results.

Citation

@misc{velingker2026camel,
  title={CAMEL: An ECG Language Model for Forecasting Cardiac Events},
  author={Velingker et al. (2026)},
  year={2026},
  note={arXiv:2602.15677}
}

• arXiv: 2602.15677