Medsyn Eval

medsyn-eval

MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models — Chen et al. (2026) (arXiv:2602.21950, 2026)

What this evaluates

Evaluates multimodal large language models on their ability to generate differential diagnoses (DDx) and select final diagnoses (FDx) for complex clinical cases. It probes cross-modal evidence calibration, testing how models weigh textual versus visual clinical evidence, and measures their sensitivity to specific evidence types.

Datasets

• MEDSYN — total ?; splits: English (-1), Chinese (-1)

Metrics

• DDx CoverageRate (%) — range: percent
  • Percentage of cases where the model's generated list of differential diagnoses fully covers the ground truth set of potential diagnoses.
• FDx SelectionAcc. (%) (primary) — range: percent
  • Percentage of cases where the model correctly selects the single ground-truth final diagnosis from the available options.
• Evidence Sensitivity — range: [0, 1]
  • Jensen-Shannon divergence between the model's answer posterior with all clinical evidence and the posterior after removing a specific evidence type: JSD(p(y|E) || p(y|E{e_m})).

Input / output format

Input: Multimodal clinical case containing up to 7 distinct clinical evidence types (e.g., medical history text, physical findings, raw diagnostic images like CT/microscopy, lab results). Evaluated in a zero-shot setting.

Output: For DDx: a list of potential diagnoses. For FDx: a single selected final diagnosis.

Scoring recipe

# DDx CoverageRate
correct_ddx = sum(1 for c in cases if set(gold_ddx[c]).issubset(set(pred_ddx[c])))
ddx_rate = correct_ddx / len(cases) * 100

# FDx SelectionAcc
correct_fdx = sum(1 for c in cases if pred_fdx[c] == gold_fdx[c])
fdx_acc = correct_fdx / len(cases) * 100

Common pitfalls

• Confusing DDx coverage (enumerating possibilities) with FDx accuracy (selecting the single correct answer), as models show a ~20pp gap between them.
• Assuming textual evidence is always more helpful; ablations show removing or randomizing text can actually improve FDx accuracy due to model bias toward text over discriminative visual evidence.
• Cross-modal misalignment: identical evidence presented as raw images vs. expert text yields different model sensitivities, often clustering off the identity line.

Evidence (verbatim from paper)

DDx CoverageRate (%) | FDx SelectionAcc. (%)

Citation

@misc{chen2026medsyn,
  title={MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2602.21950}
}

• arXiv: 2602.21950