Principlismqa Eval

principlismqa-eval

Towards Assessing Medical Ethics from Knowledge to Practice — Chang Hong et al. (2025) (arXiv:2508.05132, 2025)

What this evaluates

Evaluates large language models' ability to reason about medical ethics using the Principlism framework (autonomy, non-maleficence, beneficence, justice). It probes both theoretical knowledge of ethical principles and their practical application to complex, open-ended clinical dilemmas.

Datasets

• PrinciplismQA — total ?; splits: test (-1)

Metrics

• Knowledge accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions assessing theoretical understanding of medical ethics principles.
• Practice score (primary) — range: percent
  • Score on open-ended clinical dilemmas, evaluated via expert-validated checklists or LLM-as-a-judge (GPT-4o) scoring keypoint alignment.
• Overall score — range: percent
  • Arithmetic mean of Knowledge accuracy and Practice score.

Input / output format

Input: Multiple-choice questions for knowledge assessment; open-ended clinical ethical dilemmas for practice assessment.

Output: Single response per model per question. For multiple-choice, a selected option. For open-ended, a free-text explanation.

Scoring recipe

# Knowledge
correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
knowledge_acc = correct / len(golds)

# Practice
practice_scores = [score_response_for_dilemma(resp, checklist) for resp in responses]
practice_score = mean(practice_scores)

# Overall
overall = (knowledge_acc + practice_score) / 2

Common pitfalls

• Models often score higher on Knowledge than Practice, revealing a 'knowledge-action gap' in applying principles to dynamic scenarios.
• Domain fine-tuning can improve Practice scores but may cause forgetting of key medical ethics knowledge, lowering Knowledge accuracy.
• LLMs tend to overemphasize autonomy and non-maleficence while struggling to dynamically apply beneficence when principles conflict.

Evidence (verbatim from paper)

The overall results of PrinciplismQA evaluation are summarized in Table[6]. Among general large reasoning models, o3 achieved the highest overall score, with 74.4% Knowledge accuracy, 80.7 Practice score, and an overall score of 77.5.

Citation

@misc{hong2025towards,
  title={Towards Assessing Medical Ethics from Knowledge to Practice},
  author={Chang Hong et al. (2025)},
  year={2025},
  note={arXiv:2508.05132}
}

• arXiv: 2508.05132