# Medical Vqa Eval

> Evaluates the medical visual question answering capabilities of multimodal large language models across diverse imaging modalities and general medical knowledge domains. Use when the user wants to benchmark on VQA-RAD, SLAKE (English CLOSED), PathVQA, PMC-VQA, MMMU (Health & Medicine track), OmniMedVQA (open access), or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/medical-vqa-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/medical-vqa-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/medical-vqa-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/medical-vqa-eval

---


# medical-vqa-eval

> HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale — Chen et al. (2024) (arXiv:2406.19280, 2024)

## What this evaluates

Evaluates the medical visual question answering capabilities of multimodal large language models across diverse imaging modalities and general medical knowledge domains.

## Datasets

- **VQA-RAD** — total ?; splits: test (-1)
- **SLAKE (English CLOSED)** — total ?; splits: test (-1)
- **PathVQA** — total ?; splits: test (-1)
- **PMC-VQA** — total ?; splits: test (-1)
- **MMMU (Health & Medicine track)** — total ?; splits: test (-1)
- **OmniMedVQA (open access)** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers out of the total number of test instances. Calculated as (number of exact matches / total instances) × 100.

## Input / output format

**Input**: A medical image paired with a natural language question (VQA format).

**Output**: A text string containing the predicted answer.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if normalize_answer(pred) == normalize_answer(gold))
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Must use the English CLOSED segment for SLAKE, not the full dataset.
- All evaluations must use zero-shot inference with LLaVA's default question template.
- MMMU Health & Medicine results require official website submission rather than local evaluation.

## Evidence (verbatim from paper)

> Notably, the use of the PubMedVision led to an 11.7% increase in overall accuracy, significantly outperforming the earlier LLaVA_Med dataset.

## Citation

```bibtex
@misc{chen2024huatuogptvision,
  title={HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2406.19280}
}
```

- arXiv: 2406.19280