Ecg Expert QA Eval

ecg-expert-qa-eval

ECG-Expert-QA: A Benchmark for Evaluating Medical Large Language Models in Heart Disease Diagnosis — Wang et al. (2025) (arXiv:2502.17475, 2025)

What this evaluates

Evaluates medical large language models on heart disease diagnosis using expert-validated QA pairs. It probes clinical reasoning, risk-aware decision-making, and patient-centric interaction capabilities across multiple diagnostic sub-tasks.

Datasets

• ECG-Expert-QA — total 47211; splits: test (-1); repo https://github.com/Zaozzz/ECG-Expert-QA

Metrics

• BLEU-1 (primary) — range: [0, 1]
  • BP * p1, where p1 is modified unigram precision and BP is brevity penalty (1 if candidate length > reference length, else exp(1 - reference length / candidate length)).
• ROUGE-L — range: [0, 1]
  • F_beta = ((1+beta^2)PrecisionRecall)/(Recall+beta^2*Precision), where Precision = LCS/len(candidate), Recall = LCS/len(reference), and beta is typically 1.
• METEOR — range: [0, 1]
  • F_mean * (1 - Penalty), where F_mean = 10PR/(9P+R) and Penalty = 0.5*(chunks/matches)^3, incorporating synonym matching and stemming.
• Model-to-Model Scoring (MMS) — range: other
  • Automated scoring by a reference LLM (Model A) evaluating a target model (Model B) on semantic consistency, content coverage, and language fluency against ground truth.

Input / output format

Input: Clinical context or question related to ECG/diagnosis, formatted as a QA prompt.

Output: Natural language answer/response generated by the LLM.

Scoring recipe

def evaluate(candidates, references):
    bleu_scores = [compute_bleu1(c, r) for c, r in zip(candidates, references)]
    rouge_scores = [compute_rouge_l(c, r) for c, r in zip(candidates, references)]
    meteor_scores = [compute_meteor(c, r) for c, r in zip(candidates, references)]
    mms_scores = [llm_judge_score(c, r) for c, r in zip(candidates, references)]
    return {
        'BLEU-1': sum(bleu_scores) / len(bleu_scores),
        'ROUGE-L': sum(rouge_scores) / len(rouge_scores),
        'METEOR': sum(meteor_scores) / len(meteor_scores),
        'MMS': sum(mms_scores) / len(mms_scores)
    }

Common pitfalls

• BLEU-1 only measures unigram lexical overlap and fails to capture semantic meaning or clinical accuracy.
• Model-to-Model Scoring (MMS) relies on a single LLM as judge, which may introduce systematic bias or hallucination in clinical contexts.
• Sub-datasets vary significantly in context length, causing token-limit failures for lightweight models like MiniMind2 if not carefully filtered.

Evidence (verbatim from paper)

To evaluate the performance of LLMs on the ECG-Expert-QA dataset, we adopted four widely used evaluation metrics in natural language generation (NLG): BLEU-1*[[24]], ROUGE-L[[25]], and METEOR[[26]]*. These metrics measure the similarity between the model-generated answers and ground-truth references in terms of lexical overlap and semantic relevance.

Citation

@misc{wang2025ecgexpertqa,
  title={ECG-Expert-QA: A Benchmark for Evaluating Medical Large Language Models in Heart Disease Diagnosis},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2502.17475}
}

• arXiv: 2502.17475