Kmmmu Eval

kmmmu-eval

KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context — Lee et al. (2026) (arXiv:2604.13058, 2026)

What this evaluates

Evaluates multimodal understanding in Korean language and context across nine academic disciplines. Probes localized knowledge recall, discipline-specific conventions, and the ability to map visual and textual cues to correct answers in Korean institutional settings.

Datasets

• KMMMU — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions after converting model responses to normalized forms and comparing them to gold answers via an LLM-Judge framework. Reported as mean accuracy across three independent trials.

Input / output format

Input: Multimodal exam questions in Korean combining visual and textual modalities, presented via a shared zero-shot prompt template.

Output: Normalized answer form (e.g., multiple-choice selection or short answer) compared against gold answers.

Scoring recipe

def compute_accuracy(predictions, gold_answers, llm_judge):
    normalized = [normalize_response(p) for p in predictions]
    correct = sum(1 for p, g in zip(normalized, gold_answers) if llm_judge(p, g))
    return (correct / len(gold_answers)) * 100
# Run 3 independent trials per model, report mean and standard deviation of accuracy

Common pitfalls

• Scoring relies on an LLM-Judge framework rather than exact string matching, which can introduce judge bias or inconsistency across different model outputs.
• Proprietary models are only evaluated on the 'hard subset' due to cost constraints, making direct comparison with open-source models on the full set potentially misleading.
• Overall accuracy is averaged across nine distinct disciplines, which may mask significant performance variations in specific domains like Math or Law.

Evidence (verbatim from paper)

For scoring, model responses are first converted into normalized answer forms, and then compared with the gold answers using an LLM-Judge framework. Each model is evaluated over three independent trials, and we report mean accuracy and standard deviation.

Citation

@misc{lee2026kmmmu,
  title={KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context},
  author={Lee et al. (2026)},
  year={2026},
  note={arXiv:2604.13058}
}

• arXiv: 2604.13058