Visnumbench Eval

visnumbench-eval

VisNumBench: Evaluating Number Sense of Multimodal Large Language Models — Weng et al. (2025) (arXiv:2503.14939, 2025)

What this evaluates

Evaluates the intuitive number sense of Multimodal Large Language Models (MLLMs) by testing their ability to estimate and reason about seven visual numerical attributes (angle, scale, length, quantity, depth, area, volume) across four estimation tasks (range estimation, value estimation, value comparison, multiplicative estimation).

Datasets

• VisNumBench — total 600; splits: synthetic (300), real (300)

Metrics

• Accuracy (%) (primary) — range: percent
  • Percentage of correctly answered questions out of the total number of questions. Calculated as (number of correct predictions / total number of questions) * 100.

Input / output format

Input: Image (synthetic or real) paired with a question targeting one of seven visual numerical attributes (angle, scale, length, quantity, depth, area, volume) across four estimation tasks.

Output: Text response to the question.

Scoring recipe

correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
    total += 1
accuracy = (correct / total) * 100

Common pitfalls

• Accuracy is averaged across all tasks and attributes, but task difficulty varies significantly (e.g., quantity tasks are easiest, angle tasks are hardest).
• Performance on synthetic images does not reliably transfer to real-world images due to increased complexity and variability in the latter.
• Applying multimodal CoT or math-specialized fine-tuning does not guarantee accuracy gains and can sometimes degrade performance on real-world data.

Evidence (verbatim from paper)

We randomly selected $600$ samples ($50$ QA pairs from each numerical attribute), with $300$ sourced from VisNumBench-Synthetic and $300$ from VisNumBench-Real. Human evaluators independently answered each question and provided assessments. Accuracy (%) is reported for all experimental results, and all the results are provided in Tables[2] and [3].

Citation

@misc{weng2025visnumbench,
  title={VisNumBench: Evaluating Number Sense of Multimodal Large Language Models},
  author={Weng et al. (2025)},
  year={2025},
  note={arXiv:2503.14939}
}

• arXiv: 2503.14939