Mmfinereason Eval

mmfinereason-eval

MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods — Honglin Lin et al. (2026) (arXiv:2601.21821, 2026)

What this evaluates

Evaluates multimodal reasoning capabilities across STEM, puzzles, general VQA, and document understanding domains. Probes how well vision-language models perform on complex visual reasoning tasks under strict greedy decoding and high-resolution inference settings.

Datasets

• MMMU_val — total ?; splits: val (-1)
• MathVista_mini — total ?; splits: mini (-1)
• MathVision_test — total ?; splits: test (-1)
• MathVerse_mini — total ?; splits: mini (-1)
• Dynamath — total ?; splits: test (-1)
• LogicVista — total ?; splits: test (-1)
• VisuLogic — total ?; splits: test (-1)
• ScienceQA — total ?; splits: test (-1)
• RealWorldQA — total ?; splits: test (-1)
• MMBench-EN — total ?; splits: EN (-1)
• MMStar_test — total ?; splits: test (-1)
• AI2D_test — total ?; splits: test (-1)
• CharXiv_reas — total ?; splits: reas (-1)
• CharXiv_desc — total ?; splits: desc (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions per benchmark. Calculated as (number of correct predictions / total number of samples) × 100.

Input / output format

Input: Image-text prompts where images are resized to 2048×2048 during inference. Models are evaluated in 'thinking mode' with greedy decoding (Temperature = 0).

Output: Textual reasoning chains followed by final answers, generated via greedy decoding.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred.strip() == gold.strip())
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Using temperature > 0 during evaluation; the protocol strictly requires greedy decoding (Temperature = 0) to assess reasoning reliability.
• Evaluating at low image resolution; the protocol mandates increasing maximum resolution to 2048×2048 during inference for fair comparison.
• Comparing models not in 'thinking mode'; all baselines and proposed models must be evaluated in thinking mode to ensure consistency.

Evidence (verbatim from paper)

MMFineReason demonstrates superior data efficiency, achieving higher accuracy with significantly fewer samples and smaller model parameters compared to MMR1 and HoneyBee.

Citation

@misc{lin2026mmfinereason,
  title={MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods},
  author={Honglin Lin et al. (2026)},
  year={2026},
  note={arXiv:2601.21821}
}

• arXiv: 2601.21821