qhjqhj00 medical-vqa-eval
HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale — Chen et al. (2024) (arXiv:2406.19280, 2024)
What this evaluates
Evaluates the medical visual question answering capabilities of multimodal large language models across diverse imaging modalities and general medical knowledge domains.
Datasets
- VQA-RAD — total ?; splits: test (-1)
- SLAKE (English CLOSED) — total ?; splits: test (-1)
- PathVQA — total ?; splits: test (-1)
- PMC-VQA — total ?; splits: test (-1)
- MMMU (Health & Medicine track) — total ?; splits: test (-1)
- OmniMedVQA (open access) — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly predicted answers out of the total number of test instances. Calculated as (number of exact matches / total instances) × 100.
Input / output format
Input: A medical image paired with a natural language question (VQA format).
Output: A text string containing the predicted answer.
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if normalize_answer(pred) == normalize_answer(gold))
return (correct / len(gold_labels)) * 100
Common pitfalls
- Must use the English CLOSED segment for SLAKE, not the full dataset.
- All evaluations must use zero-shot inference with LLaVA's default question template.
- MMMU Health & Medicine results require official website submission rather than local evaluation.
Evidence (verbatim from paper)
Notably, the use of the PubMedVision led to an 11.7% increase in overall accuracy, significantly outperforming the earlier LLaVA_Med dataset.
Citation
@misc{chen2024huatuogptvision,
title={HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale},
author={Chen et al. (2024)},
year={2024},
note={arXiv:2406.19280}
}
- arXiv: 2406.19280