Heartcare Bench Eval

heartcare-bench-eval

Heartcare Suite: A Unified Multimodal ECG Suite for Dual Signal-Image Modeling and Understanding — Xie et al. (2025) (arXiv:2506.05831, 2025)

What this evaluates

Evaluates multimodal ECG understanding and clinical reasoning across closed/open question answering, report generation, and signal prediction. It probes a model's ability to align temporal signal patterns with diagnostic language and generate clinically faithful outputs.

Datasets

• Heartcare-BenchS — total ?; splits: test (-1); repo https://github.com/DCDmllm/Heartcare-Suite
• Heartcare-BenchI — total ?; splits: test (-1); repo https://github.com/DCDmllm/Heartcare-Suite

Metrics

• accuracy (primary) — range: percent
  • Computed as the percentage of correctly predicted labels for closed-ended diagnostic, waveform, and rhythm questions.
• BERTScore-F1 (F1-Bio) — range: [0, 1]
  • Measures semantic similarity between generated and reference answers using a BERT-based embedding space, reporting the F1 score.
• ROUGE-L — range: [0, 1]
  • Calculates the F1 score based on the longest common subsequence between the generated text and the reference.
• RadGraph-F1 (F1-Rad) — range: [0, 1]
  • Extracts clinical entities and relations from both generated and reference reports, then computes the F1 score over the matched graph structures.
• GPT-4-based accuracy (Acc) — range: [0, 1]
  • Uses GPT-4 to judge whether the generated report matches the reference in clinical content, outputting a binary correct/incorrect label averaged over the dataset.

Input / output format

Input: Multi-lead ECG signals (provided as raw digital time-series or converted to image form) accompanied by clinical instruction prompts.

Output: Textual responses containing diagnostic classifications, waveform/rhythm descriptions, open-ended answers, or structured clinical reports.

Scoring recipe

def compute_metrics(predictions, golds, task_type):
    if task_type == 'closed_qa':
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif task_type == 'open_qa':
        return bertscore_f1(predictions, golds), rouge_l(predictions, golds)
    elif task_type == 'report_gen':
        return gpt4_accuracy(predictions, golds), radgraph_f1(predictions, golds), rouge_l(predictions, golds)

Common pitfalls

• Baseline models often fail to process raw signal inputs, requiring conversion to images that lose temporal resolution.
• GPT-4-based accuracy penalizes semantically correct but differently phrased reports due to strict overlap requirements.
• Signal prediction tasks are excluded for baselines that cannot follow the instruction, potentially biasing cross-model comparisons.

Evidence (verbatim from paper)

Table[2] reports the results on open-ended ECG QA, evaluated using BERTScore-F1 (F1-Bio) and ROUGE-L. Tasks are divided into three subtasks: diagnosis, waveform, and rhythm.

Citation

@misc{xie2025heartcare,
  title={Heartcare Suite: A Unified Multimodal ECG Suite for Dual Signal-Image Modeling and Understanding},
  author={Xie et al. (2025)},
  year={2025},
  note={arXiv:2506.05831}
}

• arXiv: 2506.05831