Medimageedu Eval

medimageedu-eval

Rethinking Patient Education as Multi-turn Multi-modal Interaction — Yao et al. (2026) (arXiv:2604.14656, 2026)

What this evaluates

Evaluates multi-turn, multi-modal dialogue capabilities for radiology patient education, testing how well models personalize explanations based on hidden patient profiles and ground visual annotations in medical images. It probes the alignment between textual explanations and drawn/image-marked evidence, as well as safety and scope adherence in medical contexts.

Datasets

• MedImageEdu — total ?; splits: test (-1)

Metrics

• MedImageEdu Overall (primary) — range: [0, 1]
  • Composite score averaging five sub-dimensions: Consultation, Safety & Scope, Language Quality, Drawing Quality, and Image-Text Response Quality. Scores are normalized to [0, 1] and computed via an automated judge evaluating multi-turn interactions.

Input / output format

Input: Multi-turn dialogue turns containing patient questions, radiology images, and a hidden patient profile (education level, health literacy, emotional state). Models interact with a drawing tool to annotate images.

Output: Textual responses and image annotations/drawings grounded in radiology reports, generated turn-by-turn to address patient queries while adhering to safety and scope constraints.

Scoring recipe

def compute_medimageedu_score(predictions, gold):
    dims = ['Consultation', 'Safety & Scope', 'Language Quality', 'Drawing Quality', 'Image-Text Response Quality']
    dim_scores = {d: [] for d in dims}
    for turn in predictions:
        for d in dims:
            dim_scores[d].append(judge_score(turn, d))
    block_scores = {d: sum(scores)/len(scores) for d, scores in dim_scores.items()}
    overall = sum(block_scores.values()) / len(block_scores)
    return overall

Common pitfalls

• Rewarding fluent final prose while ignoring weak visual grounding or incorrect image selection.
• Assuming strong multi-turn dialogue quality implies strong multimodal grounding capabilities.
• Treating safety failures as rare edge cases rather than systematic bottlenecks across model sizes.

Evidence (verbatim from paper)

GPT-5.1 achieves the best MedImageEdu Overall score of 0.840, followed by GPT-5-mini at 0.810. Among open-weight models, Qwen3-VL-32B is strongest at 0.677, close to GPT-5-nano at 0.680, while MedGemma-27B follows at 0.606.

Citation

@misc{yao2026medimageedu,
  title={Rethinking Patient Education as Multi-turn Multi-modal Interaction},
  author={Yao et al. (2026)},
  year={2026},
  note={arXiv:2604.14656}
}

• arXiv: 2604.14656