Visulogic Eval

visulogic-eval

VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models — Weiye Xu et al. (2025) (arXiv:2504.15279, 2025)

What this evaluates

VisuLogic probes vision-centric reasoning in multimodal large language models by presenting problems that require retaining critical visual cues during image description. It eliminates text-based reasoning shortcuts, forcing models to perform genuine visual inference across categories like spatial relations, quantitative shifts, and stylistic details.

Datasets

• VisuLogic — total 1000; splits: test (1000)

Metrics

• accuracy (primary) — range: percent
  • Calculated as the percentage of correctly answered multiple-choice questions out of the total. For human baselines, it aggregates results from 100 participants each solving 10 randomly sampled questions. For models, it measures the proportion of correct selections among four options.

Input / output format

Input: For MLLMs: an image paired with a multiple-choice question. For LLMs: a GPT-4o-generated image description, the question, and four options. Three prompting modes are used: Non-CoT (direct answer), CoT (step-by-step reasoning), and Hint (CoT with solution-derived hints).

Output: Answer: \boxed{$LETTER} where $LETTER corresponds to one of the four provided options.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip() == g.strip())
    return (correct / len(gold)) * 100

Common pitfalls

• Using text-only LLMs with external image captions fails to capture essential visual details, leading to hallucinations or irrelevant responses.
• Chain-of-thought (CoT) prompting provides minimal performance gains for visual reasoning tasks compared to pure-text tasks, as current CoT training relies heavily on text corpora.
• Models struggle disproportionately with stylistic reasoning (e.g., overlays, contours) and spatial reasoning, often resorting to superficial cues like object counts.

Evidence (verbatim from paper)

We invite 100 graduate students majoring in science and engineering to solve 10 randomly sampled VisuLogic questions each, allowing 2–5 minutes per question. The aggregate accuracy over all participants constitutes the human benchmark. We apply three distinct prompting paradigms to investigate model reasoning capabilities: 1)Non-CoT prompt evaluation: Models receive a concise instruction: “Answer the question using a single word or phrase, following this format: Answer: \boxed{$LETTER}”. 2)CoT prompt evaluation: We prompt models to articulate intermediate reasoning steps: “Solve the complex visual logical reasoning problem through step-by-step reasoning. Think about the reasoning process first and answer the question following this format: Answer: \boxed{$LETTER}”.

Citation

@misc{xu2025visulogic,
  title={VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models},
  author={Weiye Xu et al. (2025)},
  year={2025},
  note={arXiv:2504.15279}
}

• arXiv: 2504.15279