Vqa Generalization Eval

vqa-generalization-eval

Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Model — Taehee Kim et al. (2024) (arXiv:2401.06400, 2024)

What this evaluates

Evaluates a model's ability to answer visual questions by generalizing from synthetic template-based training data to complex, human-written questions. It probes both closed-form accuracy and open-form reasoning capabilities across 3D-rendered and medical imaging domains.

Datasets

• CLEVR-Human — total ?; splits: val (-1)
• VQA-RAD — total ?; splits: test (-1)
• SLAKE — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered closed-form questions. Computed as the number of exact matches between predicted and ground-truth answers divided by the total number of questions.
• LAVEGPT-4 — range: [0, 1]
  • A GPT-4-based evaluation metric used to score the quality and correctness of open-form answers. Scores are averaged across all open-form instances.

Input / output format

Input: Image paired with a natural language question (closed-form or open-form).

Output: A final answer string (e.g., 'yes', 'no', or a descriptive text for open-form questions).

Scoring recipe

def compute_metrics(predictions, golds, is_open_form=False):
    if not is_open_form:
        correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
        return correct / len(golds)
    else:
        scores = [llm_evaluate(p, g) for p, g in zip(predictions, golds)]
        return sum(scores) / len(scores)

Common pitfalls

• The model is trained exclusively on synthetic/template-based QA pairs, so evaluation tests zero-shot generalization to human-written questions.
• Open-form answers are evaluated using LAVEGPT-4 rather than exact match, requiring semantic similarity assessment.
• The LLM's intermediate questions must strictly follow template formats to be understood by the underlying VQA model.

Evidence (verbatim from paper)

Table 2 summarizes the performance of the medical foundation and template-based VQA models for VQA-RAD and SLAKE. Among all the models, CoQAH reported the highest accuracy in the closed-form questions (e.g., VQA-RAD: 67.5% for CoQAH vs. 59.5% for OFA-MIMIC, SLAKE: 73.9% for CoQAH vs. 69.4% for OFA-MIMIC), and also the highest LAVEGPT-4 in the open-form questions (e.g., VQA-RAD: 0.302 for CoQAH vs. 0.274 for MedVInT-TD, SLAKE: 0.425 for CoQAH vs. 0.396 for MedVInT-TD).

Citation

@misc{kim2024coqah,
  title={Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Model},
  author={Taehee Kim et al. (2024)},
  year={2024},
  note={arXiv:2401.06400}
}

• arXiv: 2401.06400