Jmmmu Eval

jmmmu-eval

JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation — Onohara et al. (2024) (arXiv:2410.17250, 2024)

What this evaluates

This benchmark evaluates large multimodal models' ability to understand Japanese-language visual content and answer questions across multiple disciplines. It specifically probes the gap between general language translation capabilities (culture-agnostic subset) and deep cultural knowledge (culture-specific subset), revealing how models handle language variation bias and culturally grounded reasoning.

Datasets

• JMMMU — total ?; splits: CA (-1), CS (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions. For multiple-choice items, the model's predicted option letter is compared to the gold label. For open-ended items, a single word or phrase is compared to the reference answer.

Input / output format

Input: An image paired with a Japanese prompt containing either a multiple-choice question with options or an open-ended question requiring a short answer.

Output: For multiple-choice: a single letter (e.g., A, B, C, D). For open-ended: a single word or short phrase.

Scoring recipe

def score(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        extracted = parse_answer(pred)  # rule-based parser
        if extracted is None:
            extracted = random.choice(gold['options'])  # fallback
        if extracted == gold['answer']:
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Unparsable model outputs are assigned a random choice, which can artificially inflate or deflate accuracy depending on the number of options.
• The culture-agnostic (CA) subset relies on translated questions, so low scores may reflect translation artifacts or language variation bias rather than a lack of visual reasoning capability.
• The rule-based parser for extracting answers may fail on unconventional Japanese phrasing, leading to incorrect fallback assignments.

Evidence (verbatim from paper)

We present the accuracy of GPT-4o when provided only with the question text and choices, without images. Following MMMU, (i) we prepare a rule-based parser to extract the model’s choice from typical generation styles such as “答えはA” (The answer is A), making the evaluation robust to some varieties of answer styles, and (ii) when a model does not respond in a parsable format, a random choice is assigned as its answer.

Citation

@misc{onohara2024jmmmu,
  title={JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation},
  author={Onohara et al. (2024)},
  year={2024},
  note={arXiv:2410.17250}
}

• arXiv: 2410.17250