Med Mim Eval

med-mim-eval

Medical Large Vision Language Models with Multi-Image Visual Ability — Xikai Yang et al. (2025) (arXiv:2505.19031, 2025)

What this evaluates

Evaluates medical vision-language models on multi-image reasoning tasks, including temporal understanding, cross-modal comparison, multi-view diagnosis, and co-reference resolution across longitudinal and multi-modality medical imaging data.

Datasets

• Med-MIM Benchmark — total 4424; splits: held-in (3224), held-out (1200)

Metrics

• closed-type accuracy (primary) — range: [0, 1]
  • Exact match between the model's predicted answer and the ground truth for closed-type questions. Open-type questions use separate open-ended scoring protocols.

Input / output format

Input: Interleaved sequence of up to three medical images and a text question. Images are formatted as (image {id}: <Image> image embeddings </Image>).

Output: Text response answering the medical question.

Scoring recipe

def score(predictions, gold, is_closed):
    if is_closed:
        return 1.0 if predictions.strip().lower() == gold.strip().lower() else 0.0
    else:
        return open_ended_scoring(predictions, gold)

Common pitfalls

• Models must handle sequences of up to three images, not just single images.
• Composed dataset samples use location-specific prefixes (e.g., 'In the first image') that must be parsed correctly.
• Temporal tasks require chronological ordering of multi-visit images.
• Closed-type and open-type examples require different evaluation protocols.

Evidence (verbatim from paper)

Our constructed Med-MIM Benchmark comprises two parts. (a) Held-in part: To comprehensively evaluate four multi-image visual abilities, we construct the held-in Med-MIM Benchmark derived from the Med-MIM instruction dataset, which includes 2,968 closed-type examples (903, 454, 208, and 1,403 for temporal, reasoning, comparison, co-reference abilities, respectively) and 256 open-type examples (30, 30, 136, and 60, respectively).

Citation

@misc{yang2025medmim,
  title={Medical Large Vision Language Models with Multi-Image Visual Ability},
  author={Xikai Yang et al. (2025)},
  year={2025},
  note={arXiv:2505.19031}
}

• arXiv: 2505.19031