Mathagent Eval

mathagent-eval

MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detection — Yan et al. (2025) (arXiv:2503.18132, 2025)

What this evaluates

Evaluates multimodal mathematical error detection by identifying incorrect steps in student solutions and categorizing the type of error. It probes the model's ability to align visual problem elements with textual reasoning and solution paths.

Datasets

• MathAgent Evaluation Dataset — total 2500; splits: test (2500)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly identified error steps (STEP) and correctly categorized error types (CATE) out of the total instances.

Input / output format

Input: Multimodal mathematical problem (image + text) paired with a student's step-by-step solution containing an error.

Output: Identification of the specific erroneous step(s) and the corresponding error category label (e.g., VIS, MIS, CAL, REAS, KNOW).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct_step = sum(1 for p, g in zip(predictions['step'], gold['step']) if p == g)
    correct_cat = sum(1 for p, g in zip(predictions['category'], gold['category']) if p == g)
    return {
        'STEP_accuracy': correct_step / len(gold['step']) * 100,
        'CATE_accuracy': correct_cat / len(gold['category']) * 100,
        'overall_accuracy': (correct_step + correct_cat) / (2 * len(gold)) * 100
    }

Common pitfalls

• Generic visual captioning often omits crucial geometric details (e.g., edge lengths, angle measures), leading to performance drops.
• Discrepancies between flawed image transcriptions and textual problem descriptions can introduce contradictory information that harms reasoning.
• Human-level performance remains significantly higher (77%) than the best model (57%), indicating the task's inherent complexity.

Evidence (verbatim from paper)

The results demonstrate that MathAgent achieves the highest accuracy on both STEP and CATE tasks. Notably, the w/o Visual Semantic Interpreter variant exhibits the lowest performance, presumably because generic descriptions of abstract geometric images may omit crucial details like edge lengths and angle measures.

Citation

@misc{yan2025mathagent,
  title={MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detection},
  author={Yan et al. (2025)},
  year={2025},
  note={arXiv:2503.18132}
}

• arXiv: 2503.18132