Cure Eval

cure-eval

CURE: A Multimodal Benchmark for Clinical Understanding and Retrieval Evaluation — Gu et al. (2026) (arXiv:2603.19274, 2026)

What this evaluates

This benchmark evaluates multimodal large language models' ability to perform clinical differential diagnosis using patient history and medical images. It explicitly disentangles intrinsic diagnostic reasoning from external evidence retrieval by testing models under four paradigms: no context, physician-curated references, standard RAG, and agentic web search. The protocol probes how well models leverage retrieved literature versus relying on internal knowledge, and how retrieval noise impacts open-ended versus multiple-choice diagnostic accuracy.

Datasets

• CURE — total 500; splits: test (500); repo https://github.com/yanniangu/CURE

Metrics

• Hit@1 — range: [0, 1]
  • Binary indicator: 1 if the ground-truth diagnosis matches the model's top-1 ranked open-ended prediction, 0 otherwise.
• Hit@3 — range: [0, 1]
  • Binary indicator: 1 if the ground-truth diagnosis appears within the model's top-3 ranked open-ended predictions, 0 otherwise.
• MCQ (primary) — range: [0, 1]
  • Exact-match accuracy for multiple-choice diagnostic questions. 1 if the selected option matches the gold option, 0 otherwise.

Input / output format

Input: Clinical history text, corresponding medical images, and optionally external context (physician-cited abstracts, retrieved literature, or agent-searched web results).

Output: Open-ended differential diagnosis list (ranked) or a selected multiple-choice option.

Scoring recipe

def compute_metrics(predictions, gold):
    # predictions: list of top-k diagnoses or MCQ choice
    # gold: correct diagnosis or MCQ option
    hit_at_1 = 1.0 if predictions[0] == gold else 0.0
    hit_at_3 = 1.0 if gold in predictions[:3] else 0.0
    mcq_acc = 1.0 if predictions == gold else 0.0
    return {'Hit@1': hit_at_1, 'Hit@3': hit_at_3, 'MCQ': mcq_acc}

Common pitfalls

• Confusing the 'Base' evaluation setting (intrinsic reasoning only) with retrieval-augmented settings, which fundamentally alter the task from pure diagnostic reasoning to evidence retrieval and integration.
• Assuming standard RAG improves performance; the benchmark shows that uncurated retrieval often introduces 'retrieval noise' that degrades open-ended Hit@1/Hit@3 scores compared to the baseline.
• Overinterpreting MCQ stability as robustness; MCQ accuracy shows a 'ceiling effect' and is less sensitive to retrieval noise than open-ended differential diagnosis.

Evidence (verbatim from paper)

Notably, MCQ accuracy exhibits relatively minor changes under RAG, whereas open-ended differential diagnosis (Hit@1/Hit@3) degrades more consistently, implying that noise primarily harms tasks requiring deep cross-evidence integration.

Citation

@misc{gu2026cure,
  title={CURE: A Multimodal Benchmark for Clinical Understanding and Retrieval Evaluation},
  author={Gu et al. (2026)},
  year={2026},
  note={arXiv:2603.19274}
}

• arXiv: 2603.19274