Slake Med Vqa Eval

slake-med-vqa-eval

SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering — Liu et al. (2021) (arXiv:2102.09542, 2021)

What this evaluates

Evaluates medical visual question answering capabilities by testing a model's ability to reason over radiology images (CT/MRI/X-ray) to answer vision-only and knowledge-based questions in English and Chinese. It probes multimodal fusion, semantic segmentation utilization, and external medical knowledge graph integration for clinical reasoning.

Datasets

• SLAKE — total ?; splits: train (-1), test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly predicted answers out of the total number of questions. Calculated separately for vision-only (open-ended and closed-ended) and knowledge-based questions, and further broken down by language (English, Chinese).

Input / output format

Input: A radiology image (CT/MRI/X-ray) paired with a bilingual question (English or Chinese). For knowledge-based tasks, question-related embeddings from a medical knowledge graph are also provided.

Output: For closed-ended questions: a selection from multiple-choice options. For open-ended questions: free-form text answers.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100

Common pitfalls

• Open-ended questions require careful string normalization or exact matching, as free-form answers are inherently harder to evaluate than multiple-choice selections.
• The dataset stratifies questions into vision-only and knowledge-based categories; evaluating them together without stratification masks significant performance differences.
• Bilingual evaluation requires separate tokenization and embedding pipelines for English and Chinese, which can cause cross-lingual leakage if not strictly isolated.

Evidence (verbatim from paper)

We report the results for vision-only and knowledge-based questions in Table 4 and Table 5 respectively. Answers of “closed-ended” questions are limited multiple-choice options, while answers of “open-ended” questions are free-form texts. Open-ended questions are generally harder to answer than closed-ended ones. In Table 4, we report the results in accuracy for vision-only questions in both English and Chinese.

Citation

@misc{liu2021slake,
  title={SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering},
  author={Liu et al. (2021)},
  year={2021},
  note={arXiv:2102.09542}
}

• arXiv: 2102.09542