Ecg Grounding Eval

ecg-grounding-eval

ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretation — Jin et al. (2026) (arXiv:2602.04279, 2026)

What this evaluates

Evaluates a multimodal LLM's ability to perform reliable, evidence-based ECG interpretation under full and missing modality conditions. It probes diagnostic accuracy, clinical reasoning fidelity, cross-modal consistency, and real-world clinical utility compared to cardiologist standards.

Datasets

• ECG-Grounding test set — total 2381; splits: test (2381)

Metrics

• Diagnosis Accuracy (primary) — range: [0, 10] or [0, 100]
  • Rubric-based score quantifying diagnostic correctness. Graded by DeepSeek-V3.1-Terminus using predefined clinical criteria.
• Analysis Completeness — range: [0, 10] or [0, 100]
  • Rubric-based score measuring coverage of critical diagnostic elements and ECG components in the interpretation.
• Analysis Relevance — range: [0, 10] or [0, 100]
  • Rubric-based score assessing whether generated findings are clinically aligned with the final diagnosis.
• Lead Evidence Validity — range: [0, 10] or [0, 100]
  • Rubric-based score evaluating lead-specific evidence relevance rather than template-style enumeration.
• ECG Feature Grounding — range: [0, 10] or [0, 100]
  • Rubric-based score measuring grounding in verifiable ECG features.
• Evidence-Based Reasoning — range: [0, 10] or [0, 100]
  • Rubric-based score quantifying the fidelity of stepwise clinical reasoning.
• Clinical Diagnostic Fidelity — range: [0, 10] or [0, 100]
  • Rubric-based score assessing the tightness of evidence-to-diagnosis linkage.
• BLEU-4 — range: [0, 1]
  • Text-similarity metric computing n-gram precision (up to 4-grams) between time-series-only and image-only model outputs.
• ROUGE-L — range: [0, 1]
  • Text-similarity metric measuring longest common subsequence recall/precision between modality-specific outputs.
• SBERT-Score — range: [0, 1]
  • Semantic similarity score using sentence embeddings to quantify cross-modal output agreement.

Input / output format

Input: ECG time-series signals and/or ECG images (interleaved modalities), optionally with clinical prompts or protocol-guided instructions.

Output: Structured clinical interpretation containing final diagnosis, fine-grained evidence localization, lead-wise findings, and stepwise reasoning (rhythm, conduction, morphology, ischemia assessment).

Scoring recipe

def score_rubric(model_output, gold_diagnosis, rubric):
    prompt = f'Rubric: {rubric}\nGold: {gold_diagnosis}\nModel: {model_output}'
    score = llm_grader(prompt)  # DeepSeek-V3.1-Terminus returns float per dimension
    return score

def score_consistency(ts_output, img_output):
    bleu = compute_bleu_4(ts_output, img_output)
    rouge = compute_rouge_l(ts_output, img_output)
    sbert = compute_sbert_similarity(ts_output, img_output)
    return {'BLEU-4': bleu, 'ROUGE-L': rouge, 'SBERT-Score': sbert}

def score_human_eval(model_outputs, cardiologists):
    ratings = [c.score(output) for c in cardiologists for output in model_outputs]
    return mean(ratings), std(ratings)

Common pitfalls

• Non-specialized MLLMs often achieve high Analysis Completeness but low Analysis Relevance and Lead Evidence Validity due to systematic hallucinations, misleading aggregate scores.
• Cross-modal consistency relies on text-similarity metrics (BLEU/ROUGE/SBERT) which may not capture clinical nuance or factual alignment between modality-specific outputs.
• Automated grading with LLMs (e.g., DeepSeek-V3.1-Terminus) can drift from human cardiologist judgments; direct human validation is recommended for clinical deployment claims.

Evidence (verbatim from paper)

Among all non-ECG-specialized MLLMs, GPT-5.1 achieves the highest Diagnosis Accuracy. However, its absolute score remains low at 31.48, falling far short of the reliability required for real-world clinical deployment.

Citation

@misc{jin2026ecgr1,
  title={ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretation},
  author={Jin et al. (2026)},
  year={2026},
  note={arXiv:2602.04279}
}

• arXiv: 2602.04279