# Kormedmcqa Eval

> Probes large language models' ability to answer multiple-choice questions derived from South Korean healthcare professional licensing exams. It evaluates domain-specific medical knowledge, regional clinical guideline adherence, and reasoning capabilities in Korean. Use when the user wants to benchmark on KorMedMCQA, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/kormedmcqa-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/kormedmcqa-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/kormedmcqa-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/kormedmcqa-eval

---


# kormedmcqa-eval

> KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations — Kweon et al. (2024) (arXiv:2403.01469, 2024)

## What this evaluates

Probes large language models' ability to answer multiple-choice questions derived from South Korean healthcare professional licensing exams. It evaluates domain-specific medical knowledge, regional clinical guideline adherence, and reasoning capabilities in Korean.

## Datasets

- **KorMedMCQA** — total 7469; splits: test (-1), dev (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Calculated as the proportion of correctly predicted answer indices compared to the ground truth correct answer indices.

## Input / output format

**Input**: Multiple-choice question with options, formatted in a 5-shot prompt using examples from the development set.

**Output**: Model-generated text containing the predicted answer index, extracted via regular expressions.

## Scoring recipe

```python
correct = 0
for pred_idx, gold_idx in zip(predictions, gold_indices):
    if pred_idx == gold_idx:
        correct += 1
return correct / len(predictions)
```

## Common pitfalls

- Regex extraction of answer indices can fail if models output reasoning text or alternative formatting.
- 5-shot prompt selection from the development set may introduce bias or data leakage if not carefully randomized.
- Regional medical guidelines and terminology differ significantly from US benchmarks, so direct translation or English-centric training data yields poor performance.

## Evidence (verbatim from paper)

> Generated model outputs were processed using regular expressions to extract predicted answers, which were then compared to the correct answer indices.

## Citation

```bibtex
@misc{kweon2024kormedmcqa,
  title={KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations},
  author={Kweon et al. (2024)},
  year={2024},
  note={arXiv:2403.01469}
}
```

- arXiv: 2403.01469