Kokushimd 10 Eval

kokushimd-10-eval

KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations — Liu et al. (2025) (arXiv:2506.11114, 2025)

What this evaluates

This benchmark evaluates large language models' ability to reason through Japanese national healthcare licensing examinations across ten medical professions. It probes domain-specific clinical knowledge, multimodal image interpretation, and high-stakes decision-making under strict, profession-specific passing criteria.

Datasets

• KokushiMD-10 — total ?; splits: test (-1); repo https://github.com/juniorliu95/KokushiMD-10

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly answered questions. Multiple-choice questions require exact set match of selected options; partial credit is not awarded.
• official_pass_fail — range: other
  • Binary outcome: 1 if the model's exam score meets the official profession-specific threshold, 0 otherwise.

Input / output format

Input: Japanese system prompt specifying role and output format, followed by a user prompt containing the question text and, for multimodal runs, associated clinical images.

Output: Strictly formatted answer as defined in the prompt (e.g., exact option letters for multiple-choice, numerical value, or text fill-in).

Scoring recipe

def evaluate_exam(predictions, gold_answers, question_types, official_threshold):
    correct = 0
    total = len(gold_answers)
    for pred, gold, qtype in zip(predictions, gold_answers, question_types):
        if qtype == 'Multiple Choice':
            if set(pred) == set(gold): correct += 1
        else:
            if pred == gold: correct += 1
    accuracy = correct / total
    passed = accuracy >= official_threshold
    return accuracy, passed

Common pitfalls

• Multiple-choice questions use strict exact-match scoring; selecting a subset of correct options yields zero points.
• Some exams contain 'forbidden options' that trigger automatic failure if selected, overriding other correct answers.
• Passing thresholds and question weights are highly variable across professions and years, requiring profession-specific evaluation scripts.

Evidence (verbatim from paper)

Questions are categorized into four types: Single Answer, Multiple Choice, Numerical Calculation, and Blank. Each type is scored with strict correctness rules—for example, multiple-choice questions are marked incorrect unless the selected options exactly match the correct set. Scenario-based questions in nursing and midwifery are evaluated with higher weight to reflect real-world decision-making. We report per-type accuracy, as well as domain-specific thresholds aligned with official exam standards. A model is considered to have passed an exam if its score meets the official passing threshold for that exam.

Citation

@misc{liu2025kokushimd10,
  title={KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2506.11114}
}

• arXiv: 2506.11114