Vqa Gen Eval

vqa-gen-eval

VQA-GEN: A Visual Question Answering Benchmark for Domain Generalization — Unni et al. (2023) (arXiv:2311.00807, 2023)

What this evaluates

This benchmark evaluates a model's ability to generalize in Visual Question Answering under coordinated visual and textual distribution shifts. It probes robustness to image corruptions, style transfers, and linguistic variations by measuring in-domain and cross-domain accuracy.

Datasets

• VQA-GEN — total ?; splits: train (400000), test (200000)

Metrics

• Accuracy (primary) — range: percent
  • Top-1 accuracy calculated on the validation split by matching model predictions to ground-truth answers.
• MMD Loss — range: other
  • Maximum Mean Discrepancy measuring distribution shift between probability distributions. Visual features are extracted via a pre-trained ResNet-101, and textual features via BERT on 10,000 questions plus 20 syntactical features. Lower values indicate greater similarity.
• BLEU-1/2/3/4 & METEOR — range: [0, 1]
  • BLEU measures n-gram overlap between candidate and reference questions. METEOR calculates similarity using explicit word-to-word matches with recall weighted higher than precision. Used to assess question generation quality and diversity.
• Inception Score & FID — range: other
  • Inception Score evaluates image diversity (higher is better). FID (Fréchet Inception Distance) measures contextual consistency between generated and original image distributions (lower is better).

Input / output format

Input: An image and a natural language question.

Output: A single-word or short-phrase answer.

Scoring recipe

correct = 0
total = 0
for pred, gold in zip(predictions, gold_answers):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
    total += 1
accuracy = (correct / total) * 100
return accuracy

Common pitfalls

• In-domain vs cross-domain results are reported in the same table; readers must distinguish diagonal cells (in-domain) from off-diagonal cells (cross-domain transfer).
• MMD Loss requires a fixed evaluation setup: visual features from a pre-trained ResNet-101 and textual features from BERT on exactly 10,000 questions plus 20 syntactical features, not a standard VQA metric.
• BLEU and METEOR are used exclusively for evaluating question generation quality and diversity, not for scoring model answers.

Evidence (verbatim from paper)

We use these metrics for the evaluation of our experiment: Accuracy : We calculate top-1 accuracy on the validation split for each dataset by matching predictions to ground-truth answers. This evaluates model performance on the variations. MMD Loss : We utilize the Maximum Mean Discrepancy (MMD) metric for domain shift analysis, a statistical test measuring the difference between probability distributions. In this, we use a pre-trained ResNet-101 model to extract visual features, encompassing high and low-level information about the image. In the question space, BERT encodes 10,000 questions to capture semantic content, and we extract 20 syntactical features like question length. By calculating the MMD based on these features, we quantify distribution shift, with lower MMD values indicating greater similarity between distributions. Similarity Analysis : To compare the original dataset with VQA-GEN, we employed similarity metrics like BLEU-1, BLEU-2, BLEU-3, BLEU-4, and METEOR to assess the quality and diversity of the generated questions.

Citation

@misc{unni2023vqagen,
  title={VQA-GEN: A Visual Question Answering Benchmark for Domain Generalization},
  author={Unni et al. (2023)},
  year={2023},
  note={arXiv:2311.00807}
}

• arXiv: 2311.00807