qhjqhj00 geometric-problem-solving-eval
Enhancing the Geometric Problem-Solving Ability of Multimodal LLMs via Symbolic-Neural Integration — Yicheng Pan et al. (2025) (arXiv:2504.12773, 2025)
What this evaluates
This benchmark evaluates the geometric reasoning and problem-solving capabilities of multimodal large language models. It tests whether models can accurately interpret geometric diagrams and accompanying text to produce correct final answers or select the right multiple-choice option.
Datasets
- GeoQA — total 754; splits: test (754)
- Geometry3K — total 601; splits: test (601)
- PGPS9K — total 1000; splits: test (1000)
- MathVista-mini-GPS — total 208; splits: test (208)
Metrics
Top-1 accuracy(primary) — range: percent- Percentage of correctly answered problems. For the Completion setting, an LLM verifier checks if the model's final numerical answer matches the ground truth. For the Choice setting, a regular expression extracts the selected option (e.g., A, B, C, D) and compares it to the ground truth.
Input / output format
Input: Multimodal input consisting of a geometric diagram/image and a corresponding text-based problem statement or question.
Output: Either a free-form generated final answer (Completion setting) or a single letter option (A, B, C, or D) representing the selected choice (Choice setting).
Scoring recipe
def score(predictions, golds, setting='completion'):
correct = 0
for pred, gold in zip(predictions, golds):
if setting == 'completion':
is_correct = llm_verifier_check(pred, gold) # Checks last 3 sentences against gold
else:
extracted = regex_extract_option(pred)
is_correct = (extracted == gold)
if is_correct:
correct += 1
return (correct / len(predictions)) * 100
Common pitfalls
- Using Top-K accuracy (e.g., Top-3 or Top-10) overestimates MLLM performance because these models can often arrive at the correct answer after multiple attempts or sampling.
- In the Choice setting, any parsing failure during option extraction is automatically counted as an incorrect prediction.
- Evaluating on the MathVista-mini subset yields high variance due to its small sample size (208 samples).
Evidence (verbatim from paper)
We adopt Top-1 accuracy as our primary evaluation metric, rather than Top-3 or Top-10 accuracy that symbolic or neural baselines often report. Top-K evaluation strategy considers a prediction correct if any of the Top-K answers match the ground truth. However, such a metric is less suitable for MLLMs, as these models can often reach the correct answer after multiple attempts, potentially leading to an overestimation of their actual performance. In contrast, Top-1 accuracy, which requires the model to produce the correct answer in a single response, provides a more faithful reflection of its reasoning ability.
Citation
@misc{pan2025geogen,
title={Enhancing the Geometric Problem-Solving Ability of Multimodal LLMs via Symbolic-Neural Integration},
author={Yicheng Pan et al. (2025)},
year={2025},
note={arXiv:2504.12773}
}
- arXiv: 2504.12773