Mulberry Eval

mulberry-eval

Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Search — Yao et al. (2024) (arXiv:2412.18319, 2024)

What this evaluates

Evaluates the multimodal reasoning and understanding capabilities of MLLMs across diverse domains including mathematics, chart interpretation, scientific/medical images, and hallucination detection. It measures how well models generate step-by-step reasoning paths and reflect on errors to produce correct answers.

Datasets

• MathVista — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)
• ChartQA — total ?; splits: test (-1)
• DynaMath — total ?; splits: test (-1)
• HallBench — total ?; splits: test (-1)
• MM-Math — total ?; splits: test (-1)
• MMEsum — total ?; splits: test (-1)

Metrics

• Average Benchmark Score (primary) — range: percent
  • Arithmetic mean of the accuracy (or normalized score) across the 8 listed benchmarks. For MMEsum, the raw sum of sub-task scores is used directly in the average.
• MathVista — range: percent
  • Accuracy on multimodal math problems.

Input / output format

Input: A text task instruction paired with an input image.

Output: A step-by-step reasoning path followed by a final answer.

Scoring recipe

For each of the 8 benchmarks, compute the accuracy (or retrieve the provided aggregate score).
Calculate the arithmetic mean of these 8 scores to report the AVG column.
Note: MMEsum uses a raw score scale (sum of sub-task scores) rather than percentage, so it should be treated as-is when averaging.

Common pitfalls

• MMEsum reports a raw sum score (e.g., ~2000) rather than a percentage, so it should not be averaged directly with percentage-based benchmarks without normalization.
• Several baseline models have missing values ('-') in the table for certain datasets, which may skew the reported average if not handled correctly.
• The paper evaluates reasoning models but does not specify a strict format for extracting the final answer from the reasoning trace, relying on standard benchmark parsers.

Evidence (verbatim from paper)

It shows that, using the same base model LLaVA-NeXT-8B (Li et al., [2024]), our Mulberry outperforms LLaVA-Reasoner-8B and Insight-V-8B by +5.7% and +6.5% on mathematical benchmark MathVista, and by +3.0% and +1.0% on multi-disciplinary benchmark MMMU, respectively.

Citation

@misc{yao2024mulberry,
  title={Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Search},
  author={Yao et al. (2024)},
  year={2024},
  note={arXiv:2412.18319}
}

• arXiv: 2412.18319