Kvasir Vqa X1 Eval

kvasir-vqa-x1-eval

Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy — Gautam et al. (2025) (arXiv:2506.09958, 2025)

What this evaluates

Evaluates multimodal vision-language models on gastrointestinal endoscopy image understanding and clinical question answering. It probes factual recall, multi-step clinical reasoning across varying complexity levels, and robustness to realistic visual perturbations like motion blur and color shifts.

Datasets

• Kvasir-VQA-x1 — total 159549; splits: train (-1), test (-1); repo https://github.com/Simula/Kvasir-VQA-x1

Metrics

• BERT-F1 (primary) — range: [0, 1]
  • Embedding-based F1 score computed by comparing contextual BERT representations of the predicted answer and ground truth answer to assess semantic similarity.
• ROUGE-1 — range: [0, 1]
  • Unigram overlap F-score between predicted and reference text, measuring surface-level content overlap.
• LLM-Adjudicator Accuracy — range: [0, 1]
  • Binary per-category scoring (1/0) by a Qwen3-30B-A3B adjudicator comparing model output to ground truth, aggregated as the fraction of correctly addressed clinical aspects per question category.

Input / output format

Input: A single gastrointestinal endoscopy image (original or weakly augmented) paired with a natural language question string.

Output: A natural language answer string generated by the model.

Scoring recipe

def score(predictions, golds, questions, categories):
    # Standard n-gram & embedding metrics
    rouge1 = compute_rouge(predictions, golds, 'rouge1')
    bert_f1 = compute_bert_f1(predictions, golds)
    
    # LLM Adjudicator per-category accuracy
    correct = {cat: 0 for cat in categories}
    total = {cat: 0 for cat in categories}
    for pred, gold, q, cats in zip(predictions, golds, questions, categories):
        prompt = build_adjudicator_prompt(q, pred, gold, cats)
        resp = call_llm(prompt)  # Qwen3-30B-A3B
        scores = json.loads(resp)['eval_json']
        for cat in cats:
            total[cat] += 1
            if scores[cat]['score'] == 1:
                correct[cat] += 1
    cat_acc = {cat: correct[cat]/total[cat] for cat in categories}
    return {'ROUGE-1': rouge1, 'BERT-F1': bert_f1, 'Categorical_Accuracy': cat_acc}

Common pitfalls

• Relying solely on n-gram metrics (BLEU/ROUGE) which fail to capture clinical semantic correctness and multi-step reasoning depth.
• Evaluating only on the original image split, thereby missing the dataset's core robustness benchmark against visual perturbations.
• Ignoring the complexity stratification (Levels 1–3), which masks model degradation on higher-order clinical inference tasks.

Evidence (verbatim from paper)

Models are assessed using a comprehensive suite of standard VQA and natural language processing (NLP) metrics, chosen to capture various facets of response quality: ROUGE-1, ROUGE-2, ROUGE-L: These metrics measure n-gram overlap and sequence similarity, providing insights into the content overlap between the model’s answer and the ground truth. BERT-F1: An embedding-based similarity metric with F1 aggregation using BERT. This metric assesses the semantic similarity between the generated answer and the ground truth by leveraging contextual embeddings from BERT.

Citation

@misc{gautam2025kvasirvqax1,
  title={Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy},
  author={Gautam et al. (2025)},
  year={2025},
  note={arXiv:2506.09958}
}

• arXiv: 2506.09958