Jmmmu Pro Eval

jmmmu-pro-eval

JMMMU-Pro: Image-based Japanese Multi-discipline Multimodal Understanding Benchmark via Vibe Benchmark Construction — Miyai et al. (2025) (arXiv:2512.14620, 2025)

What this evaluates

Evaluates multimodal language models' ability to perform integrated visual-textual reasoning on Japanese-language tasks where questions and reference images are combined into a single composite image. It specifically probes OCR capabilities, visual perception, and cross-modal alignment in a multilingual context.

Datasets

• JMMMU-Pro — total 1320; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Standard exact-match accuracy: the proportion of correctly answered questions out of the total number of questions. Calculated as (number of correct predictions / total predictions) × 100.

Input / output format

Input: A single composite image containing both the reference visual material and the Japanese question text, requiring the model to perform joint visual perception and OCR before answering.

Output: A natural language text response containing the final answer to the question.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if normalize_answer(pred) == normalize_answer(gold):
        correct += 1
accuracy = (correct / len(gold_labels)) * 100

Common pitfalls

• Open-source models are evaluated with both Direct and Chain-of-Thought prompts, reporting the higher score, while closed-source models only use Direct prompts, creating an asymmetric evaluation setup.
• The composite image format requires strong OCR and visual perception; poor performance may stem from text extraction failures rather than reasoning deficits.
• Temperature is fixed at 0 for open-source models but uses default settings for closed-source models, which can affect output consistency and reproducibility.

Evidence (verbatim from paper)

Following MMMU-Pro [59], we evaluate the open-source LMMs with both Direct and CoT prompts (as shown in Section C), and report the higher ones in the overall results. For the closed-source LMMs, they perform reasoning regardless of the prompt types, so we evaluated them using only the Direct Prompt. ... As shown in Table 1, most open-source LMMs, except for Qwen2.5-VL-7B, show a substantial decline in accuracy on JMMMU-Pro relative to JMMMU.

Citation

@misc{miyai2025jmmmu,
  title={JMMMU-Pro: Image-based Japanese Multi-discipline Multimodal Understanding Benchmark via Vibe Benchmark Construction},
  author={Miyai et al. (2025)},
  year={2025},
  note={arXiv:2512.14620}
}

• arXiv: 2512.14620