Gmai Vl Eval

gmai-vl-eval

GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI — Li et al. (2024) (arXiv:2411.14522, 2024)

What this evaluates

Evaluates a vision-language model's ability to understand and reason over diverse medical imaging modalities (CT, MRI, X-ray, pathology slides) to answer clinical questions, diagnose diseases, and perform anatomical or lesion recognition tasks.

Datasets

• PMCVQA — total ?; splits: test (-1)
• PathVQA — total ?; splits: test (-1)
• VQA-RAD — total ?; splits: test (-1)
• SLAKE — total ?; splits: test (-1)
• OmniMedVQA — total ?; splits: test (-1)
• GMAI-MMBench — total ?; splits: val (-1), test (-1)
• MMMU Health & Medicine track — total ?; splits: val (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions out of the total number of questions in the benchmark. Reported as an overall score and broken down by clinical sub-tasks (e.g., Modality Recognition, Disease Diagnosis).

Input / output format

Input: A medical image (e.g., X-ray, CT, MRI, fundus, or pathology slide) paired with a clinical or diagnostic question.

Output: A text answer or selected option corresponding to the question.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if normalize_answer(pred) == normalize_answer(gold))
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Data leakage must be strictly prevented by using only public test sets for evaluation and verifying image MD5 hashes against training data.
• Evaluation relies on VLMEvalKit with default settings; deviations in inference parameters or prompt formatting may cause non-reproducible results.
• Benchmarks report both overall accuracy and sub-task accuracies, which should be reported separately to match the paper's breakdown.

Evidence (verbatim from paper)

Overall, with an average accuracy of 88.48%, GMAI-VL surpasses models like HuatuoGPT-Vision-34B and InternVL2-40B, establishing itself as a leading model in multimodal medical image understanding and setting a new benchmark for medical VQA tasks.

Citation

@misc{li2024gmaivl,
  title={GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2411.14522}
}

• arXiv: 2411.14522