Vqa Biomedical Eval

vqa-biomedical-eval

MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine — Yunfei Xie et al. (2024) (arXiv:2408.02900, 2024)

What this evaluates

Evaluates multimodal models on biomedical visual question answering tasks. It probes the model's ability to interpret medical images (e.g., X-rays, pathology slides) and generate accurate answers to clinical or radiological questions in both open-ended and closed-ended formats.

Datasets

• VQA-RAD — total ?; splits: test (-1)
• SLAKE — total ?; splits: test (-1)
• PathVQA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted answers out of the total number of questions. Calculated separately for open-ended and closed-ended questions, and averaged across both.

Input / output format

Input: A medical image (e.g., radiograph or pathology slide) paired with a natural language question.

Output: A natural language answer string (for open-ended) or a selected option/class label (for closed-ended).

Scoring recipe

correct = 0
total = 0
for pred, gold in zip(predictions, gold_answers):
    if normalize_text(pred) == normalize_text(gold):
        correct += 1
    total += 1
return (correct / total) * 100

Common pitfalls

• Open-ended answers are often evaluated via exact string matching, which may penalize valid paraphrases or minor formatting differences.
• Closed-ended questions are sometimes treated as classification over the training vocabulary, which can overestimate generalizability as noted in the paper's footnote.
• Fine-tuning is strictly limited to 15 epochs per dataset; deviating from this may yield different results.

Evidence (verbatim from paper)

We benchmark LLaVA-Tri on three biomedical Visual Question Answering (VQA) datasets, VQA-RAD*(Lau et al., [2018a]), SLAKE(Liu et al., [2021]), and PathVQA(He et al., [2020a])*, to assess the efficacy of aligning the model using MedTrinity-25M. Following LLaVA-Med, The model is fine-tuned for 15 epochs on each of the three VQA datasets and evaluated accordingly. As shown in [Table 3], LLaVA-Tri achieved state-of-the-art results in all of the three VQA benchmarks, with 75.0% accuracy on VQA-RAD, 87.8% on SLAKE, and 65.3% on PathVQA.

Citation

@misc{xie2024medtrinity25m,
  title={MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine},
  author={Yunfei Xie et al. (2024)},
  year={2024},
  note={arXiv:2408.02900}
}

• arXiv: 2408.02900