Polymath Eval

polymath-eval

Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark — Gupta et al. (2024) (arXiv:2410.14702, 2024)

What this evaluates

Evaluates multi-modal mathematical and cognitive reasoning capabilities on visual puzzles. It probes spatial interpretation, relational understanding, pattern recognition, and long-horizon logical reasoning using diagram-based multiple-choice questions.

Datasets

• POLYMATH — total 5000; splits: testmini (-1), test-img (-1); repo https://github.com/kevinscaria/PolyMATH

Metrics

• accuracy (primary) — range: percent
  • Exact match accuracy calculated as the number of correctly predicted options divided by the total number of questions, reported as a percentage.

Input / output format

Input: A visual puzzle (diagram/image) accompanied by a multiple-choice question. In ablation settings, the diagram is replaced with a detailed textual description.

Output: The model must generate a step-by-step solution, the final answer, and the corresponding multiple-choice option letter.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_answers):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
accuracy = (correct / len(gold_answers)) * 100

Common pitfalls

• Models significantly underperform on diagram-based questions compared to text-only descriptions, indicating poor visual comprehension rather than lack of reasoning ability.
• Accuracy varies drastically across prompting strategies (zero-shot vs. CoT vs. Step Back), so results are not directly comparable without specifying the prompt.
• Random chance baseline is not 25% but varies by category (8-16%) due to unequal option distributions or question types, making naive baselines misleading.

Evidence (verbatim from paper)

For multiple-choice questions, we use exact match for answer comparison. The model inference prompts are structured to elicit a step-by-step solution, the final answer, and the corresponding option. ... Across various prompting strategies (Table 3), Claude-3.5 Sonnet performed best with these advanced prompts, achieving up to 41.90% accuracy in Step Back Prompting, compared to 39.70% in zero shot.

Citation

@misc{gupta2024polymath,
  title={Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark},
  author={Gupta et al. (2024)},
  year={2024},
  note={arXiv:2410.14702}
}

• arXiv: 2410.14702