Grounded Ecg Understanding Eval

grounded-ecg-understanding-eval

GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images — Lan et al. (2025) (arXiv:2503.06073, 2025)

What this evaluates

Evaluates a multimodal LLM's ability to interpret 12-lead ECG signals and images, providing clinically grounded diagnoses, detailed feature annotations, and evidence-based reasoning. It also tests cardiac abnormality detection and automated report generation across multiple public ECG datasets.

Datasets

• MIMIC-IV-ECG — total 2381; splits: test (2381)
• ECG-Bench (PTB-XL, CPSC2018, G12EC, CODE-15%, CSN) — total ?; splits: test (-1)
• PTB-XL Report — total ?; splits: test (-1)
• ECG-QA — total ?; splits: test (-1)

Metrics

• DiagnosisAccuracy (primary) — range: percent
  • Evaluates whether the generated diagnosis is correct, specific, and supported by ECG findings. Results are expressed as a percentage indicating average accuracy across identified key diagnoses.
• ECGFeatureGrounding — range: [0, 100]
  • Determines if the interpretation references actual ECG features (e.g., QRS amplitude, PR interval) instead of generic terms. Scaled from 0 to 100.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve for multi-label abnormality detection.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified samples for single-label datasets (CSN, G12EC).
• Report Score — range: [0, 100]
  • Evaluates generated reports based on accuracy in rhythms, waveform descriptions, and diagnoses. Max score 100, scored by GPT-4o.

Input / output format

Input: 12-lead ECG time series signals, corresponding 12-lead ECG images, and text prompts/instructions for diagnosis, feature grounding, or report generation.

Output: Textual clinical interpretations, specific ECG feature annotations (e.g., QRS/PR intervals, lead findings), diagnostic labels, and structured ECG reports.

Scoring recipe

def score_ecg_response(response, ground_truth, task_type):
    if task_type == 'grounded_understanding':
        prompt = f'Score response against criteria: {criteria}.'
        score = gpt4o.evaluate(response, prompt)
        return score
    elif task_type == 'abnormality_detection':
        y_true = ground_truth_labels
        y_pred = model_predictions
        auc = roc_auc_score(y_true, y_pred)
        f1 = f1_score(y_true, y_pred, average='macro')
        hl = hamming_loss(y_true, y_pred)
        acc = accuracy_score(y_true, y_pred)
        return {'AUC': auc, 'F1': f1, 'HL': hl, 'Accuracy': acc}
    elif task_type == 'report_generation':
        return gpt4o.score_report(response, max_score=100)

Common pitfalls

• GPT-4o is used as an automated judge for clinical reasoning metrics, which may introduce scoring bias or variability compared to human experts.
• The cardiologist evaluation is qualitative and based on only 6 representative examples, limiting statistical generalizability.
• Training data (ECG-Grounding) is sampled from MIMIC-IV-ECG but explicitly excludes samples used to train baselines like PULSE, requiring careful split verification to avoid leakage.

Evidence (verbatim from paper)

We utilize GPT-4o to score the responses of the MLLM using a predefined set of metrics that measures the accuracy and comprehensiveness of the details provided. Specifically, these metrics include: DiagnosisAccuracy evaluates whether the generated diagnosis is correct, specific, and supported by ECG findings. Results are expressed as a percentage, indicating the average accuracy across identified key diagnoses. AnalysisCompleteness checks if all key ECG components (e.g., rhythm, intervals, waveforms, and lead-specific findings) are discussed. Results are provided in absolute terms, indicating the average number of correctly addressed key ECG features for each sample. ECGFeatureGrounding determines if the interpretation references actual ECG features (e.g., QRS amplitude, PR interval) instead of generic terms. Results are scaled from 0 to 100.

Citation

@misc{lan2025gem,
  title={GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images},
  author={Lan et al. (2025)},
  year={2025},
  note={arXiv:2503.06073}
}

• arXiv: 2503.06073