qhjqhj00 patientsim-eval
PatientSim: A Persona-Driven Simulator for Realistic Doctor-Patient Interactions — Kyung et al. (2025) (arXiv:2505.17818, 2025)
What this evaluates
Evaluates the persona fidelity, factual accuracy, and clinical plausibility of an LLM-based patient simulator in doctor-patient dialogues. It measures how well the model adheres to assigned patient profiles, maintains factual consistency, and handles out-of-profile questions plausibly.
Datasets
- PatientSim Profiles — total 170; splits: persona_eval (108), factuality_eval (52)
Metrics
Entail (%)(primary) — range: [0, 1]- Percentage of supported sentences (those related to at least one profile item) that are classified as entailment by an NLI evaluator. Calculated as the sum of entailment indicators for info-type sentences divided by the total number of info-type sentences.
Plausibility— range: [1, 4]- 4-point scale rating (1-4) assigned to unsupported sentences (those containing information not in the profile) to measure clinical plausibility.
ICov— range: [0, 1]- Proportion of profile item categories that appear in both the original profile and the LLM-derived profile from the dialogue. Calculated as the average over profiles of the ratio of overlapping items to total items.
ICon— range: [1, 4]- Semantic similarity score between original and derived profile items for overlapping categories, rated on a 4-point scale by an LLM scorer.
Persona Fidelity Score— range: [1, 4]- Average score across five criteria (Personality, Language, Recall, Confused, Realism) rated on a 4-point scale by human or LLM evaluators.
Input / output format
Input: For persona evaluation: explicit persona descriptions and generated dialogues. For factuality/plausibility: dialogue history, current sentence, and the full patient profile containing predefined items. Evaluators receive step-specific instructions and rubrics.
Output: Persona evaluation: 4-point scores per criterion. Factuality: sentence type classification, binary relation vector to profile items, and NLI label (entailment/contradiction/neutral). Plausibility: 4-point plausibility score. Dialogue-level: derived profile items and semantic similarity scores.
Scoring recipe
def compute_entail_percent(utterances, profile):
info_sents = [s for s in utterances if classify_type(s) == 'info']
supported = [s for s in info_sents if any(relate(s, item) for item in profile)]
entail_count = sum(1 for s in supported if nli_label(s, profile) == 'entailment')
return entail_count / len(supported) if supported else 0
def compute_plausibility(utterances, profile):
unsupported = [s for s in utterances if not relate(s, profile) or all(nli_label(s, profile) == 'neutral')]
scores = [score_plausibility(s) for s in unsupported] # 1-4 scale
return sum(scores) / len(scores) if unsupported else 0
def compute_icov(original_profile, derived_profile):
overlap = sum(1 for j in range(K) if original_profile[j] and derived_profile[j])
return overlap / K
def compute_icon(original_profile, derived_profile):
overlap_items = [j for j in range(K) if original_profile[j] and derived_profile[j]]
if not overlap_items: return 0
return sum(score_similarity(original_profile[j], derived_profile[j]) for j in overlap_items) / len(overlap_items)
Common pitfalls
- Confusing supported vs. unsupported sentences: supported sentences are evaluated for factual entailment/contradiction, while unsupported ones are evaluated for clinical plausibility.
- Evaluator bias: The same LLM (Gemini-2.5-Flash) is used as both the doctor role in some setups and the primary scorer, which may inflate agreement scores.
- Dialogue-level metrics depend on an LLM profile extractor; extraction errors directly propagate to ICov and ICon scores.
Evidence (verbatim from paper)
The final factual accuracy of the sentence $s^{i}_{tm}$ is represented by Entail (%), calculated as follows, ... which reflects the percentage of supported sentences that are factually accurate.
Citation
@misc{kyung2025patientsim,
title={PatientSim: A Persona-Driven Simulator for Realistic Doctor-Patient Interactions},
author={Kyung et al. (2025)},
year={2025},
note={arXiv:2505.17818}
}
- arXiv: 2505.17818