qhjqhj00 polish-medical-qa-eval
Polish-English medical knowledge transfer: A new benchmark and results — Grzybowski et al. (2024) (arXiv:2412.00559, 2024)
What this evaluates
Evaluates large language models on Polish medical licensing and specialization exams to assess cross-lingual medical knowledge transfer, domain-specific understanding, and specialty-level accuracy compared to human medical graduates.
Datasets
- Polish Medical Exams (LEK/LDEK/PES) — total 1961; splits: LEK (977), LDEK (984), PES (-1)
Metrics
score(primary) — range: other- Total number of correctly answered multiple-choice questions. Model scores are compared against human test-taker distributions (mean ± standard deviation) and percentile ranks.
Input / output format
Input: Polish-language multiple-choice medical exam questions with answer options.
Output: Model's selected answer option for each question.
Scoring recipe
def compute_score(predictions, gold):
correct = sum(1 for p, g in zip(predictions, gold) if p == g)
return correct
def rank_vs_human(model_score, human_scores):
p25, p50, p75 = np.percentile(human_scores, [25, 50, 75])
if model_score < min(human_scores): return 'below_min'
elif model_score < p25: return 'p0_25'
elif model_score < p50: return 'p25_50'
elif model_score < p75: return 'p50_75'
elif model_score < max(human_scores): return 'p75_max'
else: return 'above_max'
Common pitfalls
- Assuming exam score distributions are perfectly normal, which the authors note is an assumption rather than a proven fact.
- Small sample sizes in certain PES specializations (e.g., 6-9 participants) can heavily skew percentile comparisons and human baselines.
- Confusing cross-lingual translation fidelity with actual domain-specific medical knowledge retention.
Evidence (verbatim from paper)
The exams were taken by 33,929 participants for LEK and 4,366 for LDEK, totaling 38,295 results from medical graduates and final-year students in Poland. While all selected models pass the chosen LEK exams, only Meta-Llama-3.1-70B-Instruct and gpt-4o-2024-08-06 score within the range defined by an average number of points ± standard deviation achieved by humans.
Citation
@misc{grzybowski2024polishenglishmedical,
title={Polish-English medical knowledge transfer: A new benchmark and results},
author={Grzybowski et al. (2024)},
year={2024},
note={arXiv:2412.00559}
}
- arXiv: 2412.00559