qhjqhj00 vqa-cot-reasoning-eval
Improve Vision Language Model Chain-of-thought Reasoning — Ruohong Zhang et al. (2024) (arXiv:2410.16198, 2024)
What this evaluates
Probes the ability of vision-language models to perform multi-step chain-of-thought reasoning on visual inputs across diverse domains like charts, documents, science diagrams, and math. It measures both direct answer accuracy and structured reasoning accuracy.
Datasets
- A-OKVQA — total ?; splits: test (-1)
- ChartQA — total ?; splits: test (-1)
- DocVQA — total ?; splits: test (-1)
- InfoVQA — total ?; splits: test (-1)
- TextVQA — total ?; splits: test (-1)
- AI2D — total ?; splits: test (-1)
- ScienceQA — total ?; splits: test (-1)
- MathVista — total ?; splits: test (-1)
- OCRBench — total ?; splits: test (-1)
- MMStar — total ?; splits: test (-1)
- MMMU — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Exact-match accuracy calculated as the percentage of correctly predicted answers after extracting the final answer from the model's output.
Input / output format
Input: An image paired with a text prompt. Prompts are either direct questions or chain-of-thought (CoT) prompts requesting step-by-step reasoning.
Output: A text response. For CoT evaluation, the model must generate a reasoning trace followed by the exact pattern '# Answer: '.
Scoring recipe
def compute_accuracy(predictions, gold_answers):
correct = 0
for pred, gold in zip(predictions, gold_answers):
if '# Answer:' in pred:
pred = pred.split('# Answer:')[-1].strip()
if pred.lower() == gold.lower():
correct += 1
return (correct / len(predictions)) * 100
Common pitfalls
- For CoT evaluation, answers must be extracted specifically after the '# Answer:' pattern; failing to do so will include reasoning text in the score.
- A-OKVQA evaluation was implemented by the authors themselves, whereas other datasets use the VLMEval protocol; mixing these up causes score discrepancies.
- Direct prediction and CoT prediction should be evaluated separately, as performance varies significantly by task type (e.g., calculation tasks favor CoT, while text-rich tasks may favor direct answers).
Evidence (verbatim from paper)
When comparing model trained on direct only data (2) to that trained on format-aligned data (1), we observe an average gain of +5.6 in direct prediction accuracy (65.5→71.1) and a +2.9 improvement in CoT performance (62.7→65.6).
Citation
@misc{zhang2024improvevlmreasoning,
title={Improve Vision Language Model Chain-of-thought Reasoning},
author={Ruohong Zhang et al. (2024)},
year={2024},
note={arXiv:2410.16198}
}
- arXiv: 2410.16198