Mm Upt Eval

mm-upt-eval

First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training — Lai Wei et al. (2025) (arXiv:2505.22453, 2025)

What this evaluates

Evaluates the multi-modal mathematical reasoning capabilities of MLLMs on diverse visual math problems including geometry, charts, and tables. It tests the model's ability to solve multiple-choice and fill-in-the-blank questions using both human-created and synthetically generated unlabeled data.

Datasets

• MathVision — total ?; splits: test (-1)
• MathVerse — total ?; splits: test (-1)
• MathVista — total ?; splits: test (-1)
• We-Math — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions out of the total number of questions in the benchmark.

Input / output format

Input: Multi-modal mathematical reasoning problems containing images and text (geometry diagrams, charts, tables, multiple-choice, or fill-in-the-blank questions).

Output: Textual answer or solution to the mathematical problem.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(gold_answers)) * 100

Common pitfalls

• The paper evaluates on multiple benchmarks but reports average accuracy across them, which can mask per-benchmark performance variations.
• Training datasets (Geometry3K, GeoQA, MMR1) are used without ground-truth labels during unsupervised post-training, so evaluation relies on external benchmark labels not present during training.
• Synthetic data generation strategies (In-Context vs Direct) are compared, but the evaluation metric remains standard accuracy without checking for hallucination or factual correctness of generated questions.

Evidence (verbatim from paper)

We report accuracy (%) for each method on MathVision, MathVerse, MathVista, and We-Math.

Citation

@misc{wei2025mmupt,
  title={First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training},
  author={Lai Wei et al. (2025)},
  year={2025},
  note={arXiv:2505.22453}
}

• arXiv: 2505.22453