Kormedmcqa V Eval

kormedmcqa-v-eval

KorMedMCQA-V: A Multimodal Benchmark for Evaluating Vision-Language Models on the Korean Medical Licensing Examination — Choi et al. (2026) (arXiv:2602.13650, 2026)

What this evaluates

This benchmark evaluates vision-language models on multimodal medical reasoning using questions derived from the Korean Medical Licensing Examination. It probes the models' ability to integrate textual and visual evidence across diverse clinical imaging modalities, including cross-image reasoning when multiple scans are provided.

Datasets

• KorMedMCQA-V — total 1534; splits: test (1534); repo https://github.com/baeseongsu/kormedmcqa_v

Metrics

• accuracy (primary) — range: percent
  • Exact-match accuracy calculated as the fraction of correctly predicted option labels (A–E) out of the total number of questions. Reported as a percentage.

Input / output format

Input: A question stem, five multiple-choice answer options (A–E), and one or more associated medical images (e.g., X-ray, CT, ECG, ultrasound, endoscopy).

Output: The selected option label (A–E) formatted in JSON.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p.strip().upper() == g.strip().upper())
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Answer-position bias: The dataset has a strong frequency bias toward option 'E', as shown by the majority-label baseline scoring 22.4%.
• Multi-image degradation: Models consistently underperform on questions requiring evidence integration across 2 or 3+ images compared to single-image items.
• Modality-specific variance: Overall accuracy masks large performance gaps across imaging types (e.g., MRI/PBS vs. NST/Endoscopy), requiring per-modality analysis.

Evidence (verbatim from paper)

We score predictions by exact match between the predicted and gold option labels and report accuracy. For open-source models, we run three random seeds and report the average.

Citation

@misc{choi2026kormedmcqa,
  title={KorMedMCQA-V: A Multimodal Benchmark for Evaluating Vision-Language Models on the Korean Medical Licensing Examination},
  author={Choi et al. (2026)},
  year={2026},
  note={arXiv:2602.13650}
}

• arXiv: 2602.13650