Med Vqa Eval

med-vqa-eval

AMANDA: Agentic Medical Knowledge Augmentation for Data-Efficient Medical Visual Question Answering — Wang et al. (2025) (arXiv:2510.02328, 2025)

What this evaluates

Evaluates multimodal models on medical visual question answering across diverse imaging modalities. It probes intrinsic visual reasoning capabilities and extrinsic biomedical knowledge grounding, while measuring the model's ability to minimize clinical hallucinations.

Datasets

• VQA-RAD — total ?; splits: test (-1)
• SLAKE — total ?; splits: test (-1)
• ProbMed — total ?; splits: test (-1)

Metrics

• accuracy/recall (closed/open-ended) (primary) — range: percent
  • Accuracy is computed as the percentage of correct answers for closed-ended questions. Recall is computed for open-ended questions to measure the overlap between generated and reference answers.

Input / output format

Input: A medical image paired with a natural language question.

Output: A natural language text answer.

Scoring recipe

def compute_metrics(predictions, golds, question_types):
    correct = 0
    total = 0
    for pred, gold, qtype in zip(predictions, golds, question_types):
        if qtype == 'closed':
            if pred.strip().lower() == gold.strip().lower():
                correct += 1
        elif qtype == 'open':
            # Recall metric (exact match or token overlap variant not specified)
            if match_recall(pred, gold):
                correct += 1
        total += 1
    return (correct / total) * 100

Common pitfalls

• Using fixed-iteration reasoning instead of the proposed adaptive refinement mechanism degrades performance and increases computational cost.
• Treating open-ended and closed-ended questions identically; the protocol explicitly requires recall for open-ended and accuracy for closed-ended questions.
• Assuming more in-context examples always improve performance; the paper shows benefits plateau beyond an optimal point, making example quality more critical than quantity.

Evidence (verbatim from paper)

Following prior work, we use accuracy for closed-ended questions and recall for open-ended questions.

Citation

@misc{wang2025amanda,
  title={AMANDA: Agentic Medical Knowledge Augmentation for Data-Efficient Medical Visual Question Answering},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2510.02328}
}

• arXiv: 2510.02328