Kvasir Vqa Eval

kvasir-vqa-eval

Kvasir-VQA: A Text-Image Pair GI Tract Dataset — Gautam et al. (2024) (arXiv:2409.01437, 2024)

What this evaluates

Probes a model's ability to perform multimodal understanding and generation on gastrointestinal endoscopic images. It evaluates capabilities in descriptive captioning, answering clinical questions about visual findings, and synthesizing anatomically plausible medical images from text prompts.

Datasets

• Kvasir-VQA — total 6500; splits: train (2000), test (5000)

Metrics

• BLEU (primary) — range: [0, 1]
  • Geometric mean of modified n-gram precisions (typically up to 4-grams) with a brevity penalty to discourage overly short outputs.
• ROUGE — range: [0, 1]
  • Recall-oriented overlap metric measuring the proportion of reference n-grams that appear in the generated text.
• METEOR — range: [0, 1]
  • Weighted F-score incorporating exact matches, stems, synonyms, and paraphrases, with penalties for fragmentation.
• CIDEr — range: [0, 1]
  • TF-IDF weighted n-gram similarity score that downweights common words and emphasizes distinctive terms matching the reference.
• FID — range: other
  • Fréchet Inception Distance: squared Wasserstein distance between the multivariate Gaussian distributions of real and generated image features extracted by an Inception network.
• IS — range: other
  • Inception Score: exponential of the KL divergence between the conditional class distribution p(y|x) and the marginal distribution p(y), measuring image quality and diversity.

Input / output format

Input: Image (endoscopic GI tract) + optional text question (for VQA) or text prompt (for generation)

Output: Text string (caption or answer) or 512x512 pixel synthetic image

Scoring recipe

def compute_bleu(candidates, references):
    matches = [0]*4
    totals = [0]*4
    cand_len = sum(len(c.split()) for c in candidates)
    ref_len = min(len(r.split()) for r in references)
    for c, r in zip(candidates, references):
        c_counts = Counter(ngrams(c.split(), 4))
        r_counts = Counter(ngrams(r.split(), 4))
        for n in range(1,5):
            for gram, count in c_counts.items():
                matches[n-1] += min(count, r_counts.get(gram, 0))
            totals[n-1] += len(c.split())
    bp = math.exp(1 - ref_len/cand_len) if cand_len > ref_len else 1.0
    log_bleu = sum(math.log(matches[i]/totals[i] + 1e-9) for i in range(4)) / 4
    return bp * math.exp(log_bleu)

Common pitfalls

• The VQA dataset is synthetically generated from captions using LLaMA-3, which may introduce hallucinations or bias not present in real clinical QA.
• The image encoder is frozen during fine-tuning to save compute, which limits the model's ability to adapt visual features to the specific medical domain.
• FID and IS scores are computed using a standard Inception network not trained on medical data, potentially misrepresenting the true diagnostic quality of synthetic images.

Evidence (verbatim from paper)

We evaluated the fine-tuned model using standard captioning metrics such as Bilingual Evaluation Understudy (BLEU), Metric for Evaluation of Translation with Explicit ORdering (METEOR), and Consensus-based Image Description Evaluation (CIDEr) over 5,000 image-caption pairs from the data-subset.

Citation

@misc{gautam2024kvasirvqa,
  title={Kvasir-VQA: A Text-Image Pair GI Tract Dataset},
  author={Gautam et al. (2024)},
  year={2024},
  note={arXiv:2409.01437}
}

• arXiv: 2409.01437