qhjqhj00 biomed-vqa-eval
On Domain-Adaptive Post-Training for Multimodal Large Language Models — Cheng et al. (2024) (arXiv:2411.19930, 2024)
What this evaluates
Evaluates the domain-adaptive post-training of multimodal large language models on biomedical visual question answering tasks, measuring how well models generalize to specialized medical domains using both open and closed evaluation splits.
Datasets
- SLAKE — total ?; splits: open (-1), closed (-1)
- PathVQA — total ?; splits: open (-1), closed (-1)
- VQA-RAD — total ?; splits: open (-1), closed (-1)
- PMC-VQA — total ?; splits: open (-1), closed (-1)
Metrics
accuracy(primary) — range: percent- Standard VQA accuracy: percentage of predicted answers that exactly match or semantically align with the ground-truth answers. Reported as percentage scores in Table 2.
Input / output format
Input: Image paired with a question (instruction) in a conversational format.
Output: Text answer (precise or informative response) generated by the model.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = 0
for pred, gold_ans in zip(predictions, gold):
if normalize(pred) in [normalize(g) for g in gold_ans]:
correct += 1
return (correct / len(predictions)) * 100
Common pitfalls
- Confusing the 'open' and 'closed' evaluation splits, which have different data distributions and baseline scores.
- Failing to account for potential data contamination when using synthetic domain-adaptive training data that may overlap with test sets.
Evidence (verbatim from paper)
Table 2: Biomedicine Task Performance of general MLLMs and MLLMs after domain-adaptive post-training. The image-caption sources for AdaMLLM from PMC^Raw and AdaMLLM from PMC^Refined are PMC^Raw and PMC^Refined, respectively.
Citation
@misc{cheng2024domainadaptiveposttraining,
title={On Domain-Adaptive Post-Training for Multimodal Large Language Models},
author={Cheng et al. (2024)},
year={2024},
note={arXiv:2411.19930}
}
- arXiv: 2411.19930