Medical Diagnosis Eval

medical-diagnosis-eval

CoD, Towards an Interpretable Medical Agent using Chain of Diagnosis — Junying Chen et al. (2024) (arXiv:2407.13301, 2024)

What this evaluates

Evaluates an LLM's ability to perform medical diagnosis through interactive symptom inquiry and confidence-based decision making. It probes the model's capacity to ask targeted yes/no questions, quantify diagnostic uncertainty, and accurately predict diseases from a candidate set or open database.

Datasets

• Muzhi — total 142; splits: test (142)
• Dxy — total 104; splits: test (104)
• DxBench — total 1148; splits: test (1148)

Metrics

• Acc. (primary) — range: percent
  • Percentage of correctly predicted diseases out of total test cases.
• n — range: other
  • Average number of symptom inquiry rounds per case, capped at 5.
• H — range: other
  • Average diagnostic entropy across inquiry rounds, measuring model uncertainty.

Input / output format

Input: Chief complaints (S_exp) and implicit symptoms (S_imp). For interactive evaluation, the model receives the initial complaint and subsequent yes/no answers from a simulated patient.

Output: Disease prediction, confidence score, and optional symptom inquiry questions. For DxBench, three candidate diseases are provided as context.

Scoring recipe

def compute_metrics(predictions, gold_labels, inquiry_rounds, entropy_values):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    accuracy = (correct / len(gold_labels)) * 100
    capped_rounds = [min(r, 5) for r in inquiry_rounds]
    avg_rounds = sum(capped_rounds) / len(capped_rounds)
    avg_entropy = sum(entropy_values) / len(entropy_values)
    return {"Acc.": accuracy, "n": avg_rounds, "H": avg_entropy}

Common pitfalls

• The evaluation uses a simulated patient (GPT-4) that only answers 'yes' or 'no', which may not reflect real-world patient communication.
• Accuracy is computed over a fixed candidate set for DxBench (3 options), making it easier than open-ended diagnosis where the model must retrieve from 9,604 diseases.
• The number of inquiry rounds n is capped at 5, which may artificially limit performance on complex cases requiring more questions.

Evidence (verbatim from paper)

To evaluate diagnostic performance, we used two publicly available benchmarks: Muzhi and Dxy. DxBench includes over 1,000 real cases, covering 461 disease types from 15 departments and 5,038 symptoms. Considering the large number of diseases in DxBench, each case is provided with three candidate diseases, one of which is the ground truth. Acc. represents the accuracy of disease diagnosis (percentage). n denotes the average number of inquiry rounds, with a cap at L=5 rounds.

Citation

@misc{chen2024cod,
  title={CoD, Towards an Interpretable Medical Agent using Chain of Diagnosis},
  author={Junying Chen et al. (2024)},
  year={2024},
  note={arXiv:2407.13301}
}

• arXiv: 2407.13301