Medframeqa Eval

medframeqa-eval

MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoning — Yu et al. (2025) (arXiv:2505.16964, 2025)

What this evaluates

This benchmark evaluates multi-image medical visual question answering and clinical reasoning. It probes a model's ability to integrate diagnostic evidence across temporally coherent medical images, detect salient findings, and propagate reasoning chains to answer single-choice questions.

Datasets

• MedFrameQA — total 2851; splits: test (2851)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly answered single-choice questions out of the total number of VQA pairs. Calculated as (number of correct predictions / total questions) * 100.

Input / output format

Input: A set of 2 to 5 medical images (key-frames from instructional videos) paired with a single-choice question that requires integrating information across all provided frames.

Output: A single selected option (letter or text) corresponding to the correct answer for the multiple-choice question.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
accuracy = (correct / len(gold_labels)) * 100

Common pitfalls

• Models frequently overlook or misinterpret intermediary frames, causing reasoning errors to cascade through subsequent steps.
• Performance is highly sensitive to imaging modality and anatomical system, so averaging across all tasks may mask significant capability gaps.
• Increasing the number of frames does not linearly decrease accuracy; performance fluctuates based on visual complexity and redundancy rather than frame count alone.

Evidence (verbatim from paper)

Table 2 presents the evaluation of 10 advanced MLLMs on MedFrameQA. In general, all assessed models demonstrate persistently low accuracy, with the peak accuracy remaining below 55.00%.

Citation

@misc{yu2025medframeqa,
  title={MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoning},
  author={Yu et al. (2025)},
  year={2025},
  note={arXiv:2505.16964}
}

• arXiv: 2505.16964