visual-cot-eval
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning — Hao Shao et al. (2024) (arXiv:2403.16999, 2024)
What this evaluates
Evaluates multi-modal large language models' ability to perform chain-of-thought reasoning with dynamic visual focusing on specific image regions. It probes localized visual understanding, intermediate bounding box prediction, and multi-turn reasoning across document, chart, general VQA, relation reasoning, and fine-grained domains.
Datasets
Metrics
accuracy (primary) — range: [0, 1]
- Proportion of correctly predicted answers, reported as a decimal between 0 and 1. For qualitative/CoT evaluation, a ChatGPT-based score between 0 and 1 is also used to rate prediction accuracy.
Input / output format
Input: An image and a natural language question.
Output: A textual answer, optionally accompanied by intermediate chain-of-thought reasoning steps and predicted bounding boxes for key visual regions.
Scoring recipe
def compute_accuracy(predictions, gold_answers):
correct = 0
for pred, gold in zip(predictions, gold_answers):
if normalize_text(pred) == normalize_text(gold):
correct += 1
return correct / len(gold_answers)
Common pitfalls
- The benchmark aggregates 12 heterogeneous datasets; performance varies significantly across domains (e.g., Doc/Text vs. Fine-grained), so reporting only the average can mask domain-specific weaknesses.
- Zero-shot evaluation on SROIE, DUDE, and Visual7W uses only their test splits, which may not reflect full generalization or dataset-specific biases.
- ChatGPT-based scoring for CoT quality is subjective and highly prompt-dependent, making cross-study comparisons difficult without identical prompts.
Evidence (verbatim from paper)
We utilize 12 source datasets, as shown in Fig. 1, and when an official training/evaluation split exists, we adopt it. In cases where such a split does not exist, we randomly divide the dataset. Additionally, we incorporate the test split of SROIE, DUDE, and Visual7W to evaluate the model's zero-shot visual CoT capabilities. Following the methodology of previous MLLM studies [33, 46], we employ ChatGPT [52] and ask it to assign a numerical score between 0 and 1, where a higher score indicates better prediction accuracy.
Citation
@misc{hao2024visualcot,
title={Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning},
author={Hao Shao et al. (2024)},
year={2024},
note={arXiv:2403.16999}
}
1---2name: visual-cot-eval3description: Evaluates multi-modal large language models' ability to perform chain-of-thought reasoning with dynamic visual focusing on specific image regions. It probes localized visual understanding, intermediate bounding box prediction, and multi-turn reasoning across document, chart, general VQA, relation reasoning, and fine-grained domains. Use when the user wants to benchmark on Visual CoT Benchmark, or asks about evaluating this task. Reports accuracy.4---56# visual-cot-eval78> Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning — Hao Shao et al. (2024) (arXiv:2403.16999, 2024)910## What this evaluates1112Evaluates multi-modal large language models' ability to perform chain-of-thought reasoning with dynamic visual focusing on specific image regions. It probes localized visual understanding, intermediate bounding box prediction, and multi-turn reasoning across document, chart, general VQA, relation reasoning, and fine-grained domains.1314## Datasets1516- **Visual CoT Benchmark** — total ?; splits: test (-1); repo https://github.com/deepcs233/Visual-CoT1718## Metrics1920- `accuracy` **(primary)** — range: [0, 1]21 - Proportion of correctly predicted answers, reported as a decimal between 0 and 1. For qualitative/CoT evaluation, a ChatGPT-based score between 0 and 1 is also used to rate prediction accuracy.2223## Input / output format2425**Input**: An image and a natural language question.2627**Output**: A textual answer, optionally accompanied by intermediate chain-of-thought reasoning steps and predicted bounding boxes for key visual regions.2829## Scoring recipe3031```python32def compute_accuracy(predictions, gold_answers):33 correct = 034 for pred, gold in zip(predictions, gold_answers):35 if normalize_text(pred) == normalize_text(gold):36 correct += 137 return correct / len(gold_answers)38```3940## Common pitfalls4142- The benchmark aggregates 12 heterogeneous datasets; performance varies significantly across domains (e.g., Doc/Text vs. Fine-grained), so reporting only the average can mask domain-specific weaknesses.43- Zero-shot evaluation on SROIE, DUDE, and Visual7W uses only their test splits, which may not reflect full generalization or dataset-specific biases.44- ChatGPT-based scoring for CoT quality is subjective and highly prompt-dependent, making cross-study comparisons difficult without identical prompts.4546## Evidence (verbatim from paper)4748> We utilize 12 source datasets, as shown in Fig. 1, and when an official training/evaluation split exists, we adopt it. In cases where such a split does not exist, we randomly divide the dataset. Additionally, we incorporate the test split of SROIE, DUDE, and Visual7W to evaluate the model's zero-shot visual CoT capabilities. Following the methodology of previous MLLM studies [33, 46], we employ ChatGPT [52] and ask it to assign a numerical score between 0 and 1, where a higher score indicates better prediction accuracy.4950## Citation5152```bibtex53@misc{hao2024visualcot,54 title={Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning},55 author={Hao Shao et al. (2024)},56 year={2024},57 note={arXiv:2403.16999}58}59```6061- arXiv: 2403.16999
qhjqhj00