Unified Med Vlm Benchmark

unified-med-vlm-benchmark

MedXIAOHE: A Comprehensive Recipe for Building Medical MLLMs — Shi et al. (2026) (arXiv:2602.12705, 2026)

What this evaluates

Evaluates medical vision-language models across a broad capability surface including visual diagnosis, medical imaging, clinical reasoning, text-based QA, report generation, and instruction following. It emphasizes protocol reproducibility, deployment relevance (safety, consistency, faithfulness), and robustness to real-world clinical imagery and OCR conditions.

Datasets

• Public Med-VLM Benchmarks (30+ subsets) — total ?; splits: test (-1)
• Inhouse VQA — total 100000; splits: test (-1)
• Inhouse OCR — total ?; splits: test (-1)
• Inhouse Caption — total ?; splits: test (-1)

Metrics

• MCQ/Short QA Accuracy (primary) — range: [0, 1]
  • Binary correctness score (1.0 if deterministic option extraction or normalized numeric prediction matches gold, 0.0 otherwise).
• rubric-style structured judging — range: other
  • Structured evaluation scoring clinical correctness, completeness, and absence of unsupported claims for open-form tasks (dialogue, reports, captions).
• Key-point Coverage Score — range: other
  • Reward-and-penalty metric where correctly covered key points receive positive credit and incorrect statements are penalized.

Input / output format

Input: Multimodal inputs (clinical images, medical documents, OCR-distorted reports) paired with standardized capability-aware prompt templates. Instructions specify task type (MCQ, short QA, long QA/dialogue, report/caption, OCR, agentic search) and include auxiliary context where applicable.

Output: Deterministic option extraction for closed-form tasks; structured text outputs for open-form tasks (dialogue, reports, captions) constrained by prompt templates. For Inhouse Caption, outputs are evaluated against curated key points.

Scoring recipe

def score_instance(task_type, pred, gold):
    if task_type in ['MCQ', 'short_QA']:
        return 1.0 if normalize(pred) == gold else 0.0
    elif task_type == 'caption':
        covered = count_covered_key_points(pred, gold['key_points'])
        penalized = count_unsupported_claims(pred)
        return covered - penalized
    else:
        return rubric_judge(pred, gold, ['clinical_correctness', 'completeness', 'no_unsupported_claims'])

def evaluate_suite(predictions, golds, task_types):
    scores = [score_instance(t, p, g) for t, p, g in zip(task_types, predictions, golds)]
    return macro_average_by_category(scores)

Common pitfalls

• Cross-paper comparisons are brittle due to protocol-level mismatches (different benchmark subsets, prompting styles, scoring scripts, and data hygiene assumptions).
• Open-form task scoring is highly sensitive to protocol choices, particularly distinguishing omissions from incorrect additions in report generation.
• Real-world OCR and VQA benchmarks require handling of perspective distortion, blur, glare, and partial occlusion, which standard clean datasets do not capture.

Evidence (verbatim from paper)

Across benchmarks, we normalize evaluation with a consistent harness: Task normalization: each dataset is mapped into a small set of task families (MCQ, short QA, long QA/dialogue, report/caption generation, OCR, agentic search/decision). Prompt templates: we use capability-aware templates that standardize instruction phrasing, answer format constraints, and the placement of auxiliary context. Answer parsing: for closed-form tasks (MCQ/short QA), we enforce deterministic parsing (option extraction, numeric normalization, whitespace/punctuation normalization). For open-form tasks (dialogue/report/caption), we score with task-appropriate metrics and/or rubric-style structured judging to distinguish clinical correctness, completeness, and unsupported claims.

Citation

@misc{shi2026medxiaohe,
  title={MedXIAOHE: A Comprehensive Recipe for Building Medical MLLMs},
  author={Shi et al. (2026)},
  year={2026},
  note={arXiv:2602.12705}
}

• arXiv: 2602.12705