Ddxplus Diagnostic Reasoning Eval

ddxplus-diagnostic-reasoning-eval

Large Language Models Perform Diagnostic Reasoning — Cheng-Kuang Wu et al. (2023) (arXiv:2307.08922, 2023)

What this evaluates

Probes large language models' ability to perform iterative medical diagnostic reasoning through a simulated patient-doctor dialogue. It evaluates whether models can gather clinical evidence, generate differential diagnoses, and converge on the correct ground-truth pathology within a limited number of interaction turns.

Datasets

• DDXPlus — total ?; splits: test (-1)

Metrics

• diagnostic accuracy (primary) — range: [0, 1]
  • The proportion of instances where the model's final predicted diagnosis exactly matches the ground-truth PATHOLOGY. Computed as correct predictions divided by total test instances.

Input / output format

Input: Patient profile constructed from AGE, SEX, and INITIAL_EVIDENCE, followed by a dialogue history of clinical findings (EVIDENCES) gathered turn-by-turn.

Output: Final diagnosis prediction (PATHOLOGY), optionally preceded by a differential diagnosis list and evidence summary. Forced at turn 8 if not generated earlier.

Scoring recipe

def compute_diagnostic_accuracy(predictions, gold_pathologies):
    correct = 0
    for pred, gold in zip(predictions, gold_pathologies):
        if pred.strip().lower() == gold.strip().lower():
            correct += 1
    return correct / len(gold_pathologies)

Common pitfalls

• Ignoring the hard turn limit (T_max=8), which forces a diagnosis via an '[inform]' prefix and artificially inflates accuracy if models are allowed to continue indefinitely.
• Misclassifying domain splits: in-domain vs out-domain is defined strictly by whether the INITIAL_EVIDENCE (IE) matches the few-shot exemplars, not by random patient ID or symptom category.
• Using non-zero temperature, which introduces unnecessary randomness in a deterministic diagnostic reasoning task where the paper explicitly sets temperature to 0.

Evidence (verbatim from paper)

The AGE, SEX, IE, and EVIDENCES are used to construct the patient profile P based on a rule-based template, and the PATHOLOGY is the ground truth diagnosis for computing the diagnostic accuracy. To evaluate the diagnosis accuracy of dialogue turn $T$, if the final diagnosis has not been generated, we instruct the DSAD to make the final diagnosis by inserting a prefix "[inform]" in turn $T$'s prompt (see Appendix A.1, A.2). And to avoid lengthy dialogue, we set a pre-defined $T_{max}$ as the maximum number of turn allowed for making the final diagnosis. We determine $T_{max}$ by inviting a physician to interact with our patient bot, and find that $6\sim 8$ turns are required on average. Thus, we set $T_{max} = 8$ for our experiments.

Citation

@misc{wu2023largellmsperformdiagnosticreasoning,
  title={Large Language Models Perform Diagnostic Reasoning},
  author={Cheng-Kuang Wu et al. (2023)},
  year={2023},
  note={arXiv:2307.08922}
}

• arXiv: 2307.08922