Visonlyqa Eval

visonlyqa-eval

VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information — Kamoi et al. (2024) (arXiv:2412.00947, 2024)

What this evaluates

This benchmark probes a model's ability to accurately perceive basic geometric information—such as shape, angle, length, area, and intersections—in scientific figures and diagrams. It isolates visual perception from higher-level reasoning or domain knowledge by using direct, low-reasoning questions on synthetic and real-world images.

Datasets

• VisOnlyQA — total ?; splits: train (10000), Eval-Real (-1), Eval-Synthetic (-1); repo https://github.com/psunlpgroup/VisOnlyQA

Metrics

• accuracy (primary) — range: percent
  • The percentage of correctly answered questions out of the total number of instances. Calculated as (number of exact matches between model prediction and ground truth label) / (total number of predictions) * 100.

Input / output format

Input: A single image (scientific figure, geometric shape, chart, or 3D diagram) paired with a natural language question asking about its geometric properties.

Output: A text response containing the model's answer. Models may optionally generate chain-of-thought reasoning before the final answer.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Assuming chain-of-thought prompting improves performance; the paper shows CoT does not consistently help because reasoning is not the bottleneck.
• Misattributing low scores to reasoning or knowledge deficits; error analysis confirms that almost all mistakes stem from visual perception errors rather than logical flaws.

Evidence (verbatim from paper)

Table 5 shows the accuracy of LVLMs on Eval-Real and Eval-Synthetic (with no chain-of-thought). The performance of LVLMs is far from perfect on all tasks, with the best average accuracies of $79.0%$ and $55.4%$ by Gemini 2.5 Pro on the Real and Synthetic splits, while human performance is nearly perfect $(93.5%$ and $95.0%)$ .

Citation

@misc{kamoi2024visonlyqa,
  title={VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information},
  author={Kamoi et al. (2024)},
  year={2024},
  note={arXiv:2412.00947}
}

• arXiv: 2412.00947