qhjqhj00 rjua-sps-clinical-eval
Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm — Liu et al. (2024) (arXiv:2403.16446, 2024)
What this evaluates
Evaluates LLMs' clinical capabilities across single-turn QA, diagnostic reasoning, and multi-turn dialogue using a urology-specific standardized patient dataset. It probes information gathering, diagnostic logic, treatment planning, and adherence to clinical workflows.
Datasets
- RJUA-SPs — total ?; splits: test (-1)
Metrics
Diagnosis Accuracy(primary) — range: percent- Percentage of correct diagnoses compared to ground-truth retrieved via RAE from Standardized Patients.
Treatment Accuracy— range: percent- Percentage of correct treatment recommendations compared to ground-truth retrieved via RAE.
Diagnostic Reasoning Accuracy— range: percent- Percentage of correctly recommended laboratory tests and examinations compared to ground-truth.
Clinical Capability Scores— range: score- Scores for Information Completeness, Behavior Standardization, Guidance Rationality, Diagnostic Logicality, Treatment Logicality, and Clinical Applicability, calculated by matching dialogue turns against clinical practice pathways (LCP) via RAE.
Input / output format
Input: Patient initial query or symptoms (single-turn); dialogue history with standardized patient responses (multi-turn).
Output: Model-generated medical advice, diagnosis, treatment recommendations, or dialogue turns simulating a doctor.
Scoring recipe
def score_model(model_output, task_type):
gt = retrieve_ground_truth_from_SPs(task_type)
if task_type in ['single_turn_qa', 'diagnostic_reasoning']:
return 100.0 if exact_match(model_output, gt) else 0.0
elif task_type == 'multi_turn_dialogue':
capability_scores = {}
for capability in ['Information Completeness', 'Guidance Rationality', 'Diagnostic Logicality', 'Treatment Logicality', 'Clinical Applicability']:
matches = sum(1 for turn in model_output if match_clinic_pathway(turn, gt, capability))
capability_scores[capability] = (matches / len(model_output)) * 100
return capability_scores
return 0.0
Common pitfalls
- Models frequently hallucinate unnecessary medical tests not indicated by ground-truth clinical pathways.
- LLMs often fail to follow the standard clinical workflow order for collecting patient information, leading to low Behavior Standardization scores.
- Performance drops significantly in zero-shot settings where models lack complete medical context.
Evidence (verbatim from paper)
Single-turn Clinical QA. We calculate the diagnosis and treatment accuracy, where RAE can retrieve the ground-truth from SPs. Medical Diagnostic Reasoning. We calculate the accuracy of the recommended laboratory test and examination, where RAE can retrieve the ground-truth from SPs. Multi-turn Diagnostic Dialogue. Importantly, RAE can measure the clinical capabilities of LLMs by the retrieval task.
Citation
@misc{liu2024towardsautomatic,
title={Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm},
author={Liu et al. (2024)},
year={2024},
note={arXiv:2403.16446}
}
- arXiv: 2403.16446