Kormedmcqa Eval

kormedmcqa-eval

KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations — Kweon et al. (2024) (arXiv:2403.01469, 2024)

What this evaluates

Probes large language models' ability to answer multiple-choice questions derived from South Korean healthcare professional licensing exams. It evaluates domain-specific medical knowledge, regional clinical guideline adherence, and reasoning capabilities in Korean.

Datasets

• KorMedMCQA — total 7469; splits: test (-1), dev (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Calculated as the proportion of correctly predicted answer indices compared to the ground truth correct answer indices.

Input / output format

Input: Multiple-choice question with options, formatted in a 5-shot prompt using examples from the development set.

Output: Model-generated text containing the predicted answer index, extracted via regular expressions.

Scoring recipe

correct = 0
for pred_idx, gold_idx in zip(predictions, gold_indices):
    if pred_idx == gold_idx:
        correct += 1
return correct / len(predictions)

Common pitfalls

• Regex extraction of answer indices can fail if models output reasoning text or alternative formatting.
• 5-shot prompt selection from the development set may introduce bias or data leakage if not carefully randomized.
• Regional medical guidelines and terminology differ significantly from US benchmarks, so direct translation or English-centric training data yields poor performance.

Evidence (verbatim from paper)

Generated model outputs were processed using regular expressions to extract predicted answers, which were then compared to the correct answer indices.

Citation

@misc{kweon2024kormedmcqa,
  title={KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations},
  author={Kweon et al. (2024)},
  year={2024},
  note={arXiv:2403.01469}
}

• arXiv: 2403.01469