We Math Eval

we-math-eval

We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning? — Qiao et al. (2024) (arXiv:2407.01284, 2024)

What this evaluates

Evaluates the visual mathematical reasoning capabilities of Large Multimodal Models (LMMs). It probes their ability to decompose composite problems, apply hierarchical knowledge concepts, and reason through multi-step visual math tasks without relying on rote memorization.

Datasets

• We-Math testmini — total 1740; splits: testmini (1740); repo https://github.com/We-Math/We-Math

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted options after regex matching against ground truth.
• four-dimensional metric (IK, IG, CM, RM) — range: percent
  • Classifies model reasoning into Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery (CM), or Rote Memorization (RM). Reported as percentages under strict and loose evaluation settings.

Input / output format

Input: Multiple-choice visual mathematical problems accompanied by an extra "uncertain" option to mitigate option-deduction.

Output: Predicted option letter or answer string.

Scoring recipe

# Accuracy
pred = extract_regex(model_output)
score = 1.0 if pred == ground_truth else 0.0

# Four-dimensional metric
# Classify reasoning trace into:
# IK: Insufficient Knowledge (fails to identify core concept)
# IG: Inadequate Generalization (knows concept but fails application)
# CM: Complete Mastery (correct reasoning & answer)
# RM: Rote Memorization (matches training without reasoning)
# Compute category percentages under strict/loose settings.

Common pitfalls

• Models often bypass reasoning by deducing answers directly from multiple-choice options.
• Performance drops sharply as problem complexity (number of knowledge concepts/steps) increases.
• Visual measurement tasks (e.g., angles, units) are disproportionately difficult due to visual encoder limitations.

Evidence (verbatim from paper)

For automated evaluation, we standardize all samples into a multiple-choice format. We use regex to match the LMMs’ predictions and then calculate their accuracy against the ground-truth answers for main results. For analyses in section [3.2] and [3.3], we utilize the four-dimensional metric described in section [2.2] for assessment.

Citation

@misc{qiao2024wemath,
  title={We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?},
  author={Qiao et al. (2024)},
  year={2024},
  note={arXiv:2407.01284}
}

• arXiv: 2407.01284