Gpro Eval

gpro-eval

Addressing Overthinking in Large Vision-Language Models via Gated Perception-Reasoning Optimization — Diao et al. (2026) (arXiv:2601.04442, 2026)

What this evaluates

Evaluates large vision-language models on complex mathematical and visual reasoning tasks, measuring both correctness and computational efficiency. It specifically probes the model's ability to avoid excessive chain-of-thought generation (overthinking) by dynamically routing computation between fast perception, slow perception, and slow reasoning paths.

Datasets

• MathVision — total ?; splits: test (-1)
• MathVerse — total ?; splits: test (-1)
• MathVista — total ?; splits: test (-1)
• DynaMath — total ?; splits: test (-1)
• MM-Vet — total ?; splits: test (-1)

Metrics

• accuracy (%) (primary) — range: percent
  • Percentage of questions answered correctly out of the total number of questions in the benchmark.
• average response length (tokens) — range: other
  • Mean number of generated tokens per question across the benchmark test set.

Input / output format

Input: Multimodal input consisting of one or more images paired with a text prompt describing a mathematical, geometric, or spatial reasoning problem.

Output: Natural language text containing step-by-step reasoning (optional) and a final answer.

Scoring recipe

def compute_metrics(predictions, gold_answers, token_counts):
    correct = sum(1 for pred, gold in zip(predictions, gold_answers) if pred == gold)
    accuracy = (correct / len(gold_answers)) * 100
    avg_tokens = sum(token_counts) / len(token_counts)
    return {'accuracy (%)': accuracy, 'average response length (tokens)': avg_tokens}

Common pitfalls

• Assuming longer chain-of-thought generation always improves accuracy, whereas the paper shows it often introduces noise and hallucination.
• Failing to distinguish between perceptual failures (misreading visual elements) and reasoning failures (logical errors), which require different mitigation strategies.
• Evaluating efficiency solely on wall-clock time without accounting for token generation length, which directly correlates with computational cost in autoregressive models.

Evidence (verbatim from paper)

Table[2] presents a comprehensive comparison across five multimodal reasoning benchmarks. We report both accuracy (%) and average response length (tokens) to facilitate a dual analysis of performance and computational efficiency.

Citation

@misc{diao2026gpro,
  title={Addressing Overthinking in Large Vision-Language Models via Gated Perception-Reasoning Optimization},
  author={Diao et al. (2026)},
  year={2026},
  note={arXiv:2601.04442}
}

• arXiv: 2601.04442