qhjqhj00 vg-cot-eval
VG-CoT: Towards Trustworthy Visual Reasoning via Grounded Chain-of-Thought — Lim et al. (2026) (arXiv:2604.21396, 2026)
What this evaluates
Evaluates the visual reasoning and grounding capabilities of Large Vision-Language Models (LVLMs) by measuring the quality of their step-by-step rationales, the accuracy of their final answers, and the alignment between the generated reasoning and the prediction.
Datasets
- VG-CoT — total ?; splits: test (-1)
Metrics
Rationale Quality (RQ)(primary) — range: percent- Composite score assessing visual evidence utilization, logical coherence, and reasoning completeness of the generated chain-of-thought.
Answer Accuracy (AA)(primary) — range: percent- Overall correctness of the model's final answer, calculated as the percentage of exact matches against ground truth.
Reasoning-Answer Alignment (RAA)(primary) — range: percent- Measures the consistency and faithfulness between the generated rationale and the final prediction, typically via LLM-judgment or logical entailment.
mAP@0.5 / mAP@0.75— range: percent- Mean Average Precision for bounding box localization of visual evidence against pseudo-label ground truths at IoU thresholds of 0.5 and 0.75.
Input / output format
Input: An image and a natural language question.
Output: A step-by-step reasoning process explicitly grounded in visual evidence, followed by a final answer.
Scoring recipe
def compute_metrics(predictions, golds):
aa = sum(1 for p, g in zip(predictions['answers'], golds['answers']) if p == g) / len(predictions['answers'])
raa = consistency_score(predictions['rationales'], predictions['answers'])
rq = composite_score(predictions['rationales'], golds['evidence'])
mAP = mean_average_precision(predictions['boxes'], golds['boxes'], iou_thresh=0.5)
return {'AA': aa, 'RAA': raa, 'RQ': rq, 'mAP@0.5': mAP}
Common pitfalls
- Visual evidence grounding scores are consistently lower than text-based reasoning scores, indicating spatial grounding is inherently harder than logical generation.
- Alignment anomalies can occur where fine-tuning improves overall accuracy but slightly decreases consistency between rationale and answer.
- Scene-text tasks pose a significant bottleneck for precise spatial grounding compared to relational or general VQA tasks.
Evidence (verbatim from paper)
The models are evaluated across three core dimensions. First, Rationale Quality (RQ) assesses visual evidence utilization, logical coherence, and reasoning completeness. Second, Answer Accuracy (AA) represents the overall correctness of the model’s output. Finally, Reasoning-Answer Alignment (RAA) measures the consistency and faithfulness between the generated rationale and the final prediction.
Citation
@misc{lim2026vgcot,
title={VG-CoT: Towards Trustworthy Visual Reasoning via Grounded Chain-of-Thought},
author={Lim et al. (2026)},
year={2026},
note={arXiv:2604.21396}
}
- arXiv: 2604.21396