Dynamicare Medical Diagnosis Eval

dynamicare-medical-diagnosis-eval

DynamiCare: A Dynamic Multi-Agent Framework for Interactive and Open-Ended Medical Decision-Making — Shang et al. (2025) (arXiv:2507.02616, 2025)

What this evaluates

Evaluates a dynamic multi-agent framework's ability to perform interactive, open-ended medical diagnosis by querying patients and ranking potential diagnoses. It also assesses the system's performance on interactive multiple-choice medical QA and the quality of simulated patient responses.

Datasets

• MIMIC-Patient — total 500; splits: test (500)
• MEDIQ — total 340; splits: test (340)

Metrics

• Hit@K (primary) — range: [0, 1]
  • Binary indicator (1 if at least one ground-truth diagnosis appears in the top-K predicted diagnoses, 0 otherwise). Averaged over the dataset.
• Rec@K — range: [0, 1]
  • Proportion of all ground-truth diagnoses found within the top-K predicted diagnoses. Averaged over the dataset.
• Accuracy — range: [0, 1]
  • Exact match rate for multiple-choice questions on the MEDIQ benchmark.
• Ave-Q — range: other
  • Average number of questions asked by the doctor agent per patient case.
• Truthfulness — range: [0, 2]
  • Human-annotated score (0-2 scale) measuring consistency between patient responses and the ground-truth JSON record.
• Relevance — range: [0, 2]
  • Human-annotated score (0-2 scale) measuring how directly and adequately the patient response addresses the doctor's question.

Input / output format

Input: Interactive dialogue history with patient records or static JSON patient profiles; for MEDIQ, multiple-choice clinical questions.

Output: Ranked list of up to 10 predicted diagnoses (mapped to ICD-9 codes); for MEDIQ, a single selected option; for Patient System, a natural language response.

Scoring recipe

def score_hit_rec(predictions, ground_truth):
    # predictions: list of up to 10 predicted ICD-9 codes
    # ground_truth: list of ground truth ICD-9 codes
    # Match rule: first 3 digits must match
    pred_3digit = set(code[:3] for code in predictions)
    gt_3digit = set(code[:3] for code in ground_truth)
    hit = 1.0 if pred_3digit & gt_3digit else 0.0
    rec = len(pred_3digit & gt_3digit) / len(gt_3digit) if gt_3digit else 0.0
    return hit, rec

Common pitfalls

• ICD-9 matching uses only the first 3 digits (high-level category), not full 5-digit codes, which significantly affects Hit/Rec scores.
• Ground truth typically contains 1-5 correct labels per patient, so Hit@K and Rec@K behave differently than in standard single-label retrieval benchmarks.
• MEDIQ evaluation uses only 200 randomly sampled cases from iMedQA, not the full dataset, so results are not directly comparable to full-dataset baselines.

Evidence (verbatim from paper)

We then evaluate the results using the top-k hit rate (Hit@5, Hit@10) and recall (Rec@5, Rec@10), which measure the system’s ability to rank correct diagnoses among its predictions. As the doctor agent is prompted to return a list of up to 10 likely diagnoses, and the ground truth typically includes between 1 to 5 correct labels, Hit@K metrics help assess how well the system captures correct diagnoses within a ranked list, whereas Rec@K offers a perspective on how many ground truth diagnoses are successfully identified.

Citation

@misc{shang2025dynamicare,
  title={DynamiCare: A Dynamic Multi-Agent Framework for Interactive and Open-Ended Medical Decision-Making},
  author={Shang et al. (2025)},
  year={2025},
  note={arXiv:2507.02616}
}

• arXiv: 2507.02616