Ddxplus Eval

ddxplus-eval

DDXPlus: A New Dataset For Automatic Medical Diagnosis — Fansi Tchango et al. (2022) (arXiv:2205.09148, 2022)

What this evaluates

Evaluates an agent's ability to iteratively collect clinical evidence and generate a ranked list of differential diagnoses for a simulated patient. It probes the system's diagnostic reasoning, evidence-gathering efficiency, and alignment with ground-truth pathologies.

Datasets

• DDXPlus — total ?; splits: test (-1)

Metrics

• DDF1 (primary) — range: percent
  • Harmonic mean of differential diagnosis recall (DDR) and precision (DDP). Measures the overall accuracy of the predicted differential set against the ground truth.
• DDR — range: percent
  • Recall of the predicted differential diagnosis set against the ground truth set.
• DDP — range: percent
  • Precision of the predicted differential diagnosis set against the ground truth set.
• GTPA — range: percent
  • Accuracy of including the ground truth pathology anywhere in the predicted differential diagnosis list.
• GTPA@1 — range: percent
  • Accuracy of the ground truth pathology being ranked as the top-1 prediction.
• PER — range: percent
  • Recall of positively collected evidence during the interaction.

Input / output format

Input: Patient demographics (age, sex) and an initial piece of clinical evidence. The model then engages in an iterative dialogue, querying symptoms or antecedents up to a maximum of 30 turns.

Output: A ranked probability distribution over pathologies representing the predicted differential diagnosis at the end of the interaction.

Scoring recipe

def compute_metrics(predicted_differential, gold_differential, gold_pathology, collected_evidence, gold_evidence):
    # Post-process: filter pathologies with mass <= 0.01
    pred_set = {p for p, mass in predicted_differential.items() if mass > 0.01}
    gold_set = {p for p, mass in gold_differential.items() if mass > 0.01}
    
    # Differential metrics
    tp = len(pred_set & gold_set)
    ddr = (tp / len(gold_set) * 100) if gold_set else 0.0
    ddp = (tp / len(pred_set) * 100) if pred_set else 0.0
    ddf1 = (2 * ddr * ddp / (ddr + ddp)) if (ddr + ddp) > 0 else 0.0
    
    # Ground truth pathology accuracy
    gtpa = (100.0 if gold_pathology in pred_set else 0.0)
    gtpa_at_1 = (100.0 if predicted_differential[0] == gold_pathology else 0.0)
    
    # Evidence collection recall
    per = (len(collected_evidence & gold_evidence) / len(gold_evidence) * 100) if gold_evidence else 0.0
    
    return {'DDR': ddr, 'DDP': ddp, 'DDF1': ddf1, 'GTPA': gtpa, 'GTPA@1': gtpa_at_1, 'PER': per}

Common pitfalls

• Applying GTPA@1 to models trained to predict the full differential diagnosis, as the ground truth pathology is not guaranteed to be the top-ranked entry.
• Measuring evidence precision instead of recall (PER), since asking negative questions is clinically valid and expected.
• Failing to apply the 0.01 probability mass threshold when computing differential diagnosis metrics, which inflates metric values with highly unlikely pathologies.

Evidence (verbatim from paper)

An AD system is typically tasked to collect (i) relevant evidences from a patient, (ii) make accurate predictions regarding the patient's differential, and (iii) operate in a minimum number of turns. As such, we report on the interaction length (IL), and evaluate the evidence collection by measuring the recall (PER). We do not measure the evidence precision as it is sometimes necessary to ask negative questions. Additionally, we calculate the recall (DDR), precision (DDP) and F1 score (DDF1) of the differentials. Finally, we report the accuracy of the inclusion of the ground truth pathology (i.e., the pathology a patient was simulated from) in the predicted differential diagnosis (GTPA@1 and GTPA).

Citation

@misc{fansi_tcango2022ddxplus,
  title={DDXPlus: A New Dataset For Automatic Medical Diagnosis},
  author={Fansi Tchango et al. (2022)},
  year={2022},
  note={arXiv:2205.09148}
}

• arXiv: 2205.09148