Mathverse Eval

mathverse-eval

MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? — Renrui Zhang et al. (2024) (arXiv:2403.14624, 2024)

What this evaluates

This benchmark evaluates the visual mathematical reasoning capabilities of multi-modal large language models (MLLMs), specifically probing whether they genuinely interpret geometric diagrams or merely rely on textual redundancy. It measures performance across different problem formulations (varying text/image ratios) and mathematical subjects like plane geometry, solid geometry, and functions.

Datasets

• MATHVERSE — total 15672; splits: testmini (4728), test (10944)

Metrics

• accuracy (primary) — range: percent
  • Binary correctness of the final predicted answer against the ground truth, averaged across all instances. Reported as a percentage.
• CoT evaluation score — range: [0, 1]
  • Scores intermediate reasoning steps (0 or 1 per step). Multi-step score is the average of step scores. Final score combines multi-step score (weight 0.7) and final answer correctness (weight 0.3).

Input / output format

Input: Multi-modal input consisting of a mathematical diagram image and a corresponding text question. The benchmark includes six problem versions that manipulate the presence of text and visual elements (e.g., Text Dominant, Text Only, Vision-only).

Output: Model generates a step-by-step Chain-of-Thought reasoning process followed by a final answer (multiple-choice selection or free-form text).

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return (correct / len(golds)) * 100

def compute_cot_score(steps, final_preds, golds):
    step_scores = [1 if step_correct else 0 for step_correct in steps]
    multi_step = sum(step_scores) / len(step_scores)
    final_correct = sum(1 for p, g in zip(final_preds, golds) if p == g)
    final_score = final_correct / len(golds)
    return multi_step * 0.7 + final_score * 0.3

Common pitfalls

• Models often achieve high scores by exploiting textual redundancy in the problem descriptions rather than actually interpreting the diagram, leading to inflated performance when visual input is removed.
• The CoT evaluation score can mask final answer errors by rewarding correct intermediate reasoning steps, creating a significant discrepancy between CoT-E and standard accuracy metrics.
• Evaluations are strictly zero-shot; using few-shot prompting or fine-tuning on the benchmark will invalidate the reported baseline comparisons.

Evidence (verbatim from paper)

In the subsequent experiments, all quantitative results are assessed using the testmini subset of MATHVERSE. ... The highest accuracy for closed-source and open-source MLLMs is marked in red and blue respectively.

Citation

@misc{zhang2024mathverse,
  title={MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?},
  author={Renrui Zhang et al. (2024)},
  year={2024},
  note={arXiv:2403.14624}
}

• arXiv: 2403.14624