Mammoth Vl Multimodal Eval

mammoth-vl-multimodal-eval

MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale — Jarvis Guo et al. (2024) (arXiv:2412.05237, 2024)

What this evaluates

Evaluates multimodal reasoning and instruction-following capabilities across single-image, multi-image, and video scenarios. Probes OCR, chart/document understanding, mathematical reasoning, and real-world visual interactions.

Datasets

• AI2D — total ?; splits: test (-1)
• ChartQA — total ?; splits: test (-1)
• DocVQA — total ?; splits: test (-1)
• InfoVQA — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)
• MMMU-Pro — total ?; splits: test (-1)
• SeedBench — total ?; splits: test (-1)
• MMBench — total ?; splits: test (-1)
• MMvet — total ?; splits: test (-1)
• Mathverse — total ?; splits: test (-1)
• Mathvista — total ?; splits: test (-1)
• RealworldQA — total ?; splits: test (-1)
• WildVision — total ?; splits: test (-1)
• Llava-Wilder-Small — total ?; splits: test (-1)
• MuirBench — total ?; splits: test (-1)
• MEGABench — total ?; splits: test (-1)
• EgoSchema — total ?; splits: test (-1)
• PerceptionTest — total ?; splits: test (-1)
• SeedBench (Video) — total ?; splits: video (-1)
• MLVU — total ?; splits: test (-1)
• MVBench — total ?; splits: test (-1)
• VideoMME — total ?; splits: w/o subs (-1)

Metrics

• Benchmark Score (primary) — range: percent
  • Percentage score reported per benchmark. Varies by dataset (e.g., exact match for QA, F1 for VQA, or custom scoring for reasoning). All results are aggregated and reported as percentages in the tables.

Input / output format

Input: Image(s) or video frames paired with natural language instructions/prompts.

Output: Text response generated via greedy decoding.

Scoring recipe

# Pseudo-code for evaluation protocol
for benchmark in benchmarks:
    model_output = model.generate(prompt, image, greedy=True, zero_shot=True)
    score = benchmark.evaluate(model_output, gold)  # Uses benchmark-specific metric
    results[benchmark] = score * 100  # Reported as percentage
return results

Common pitfalls

• Using non-zero-shot or non-greedy decoding settings, which deviates from the reported evaluation protocol.
• Assuming a single uniform metric across all 23 benchmarks; each benchmark uses its own specific scoring method.
• Ignoring split variations (e.g., 'dev' vs 'test' vs 'w/o subs') explicitly noted in the benchmark tables.

Evidence (verbatim from paper)

We conduct standardized, reproducible evaluations of our model across all 23 benchmarks using LMMs-Eval(Zhang et al., [2024a]). To ensure a fair comparison with other MLLMs, we primarily report results from the original papers. When results are unavailable, we onboard the models in LMMs-Eval and evaluate them using consistent settings. All results are reported using greedy decoding and zero-shot settings unless specified.

Citation

@misc{guo2024mammothvl,
  title={MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale},
  author={Jarvis Guo et al. (2024)},
  year={2024},
  note={arXiv:2412.05237}
}

• arXiv: 2412.05237