3mdbench Eval

3mdbench-eval

3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark — Sviridov et al. (2025) (arXiv:2504.13861, 2025)

What this evaluates

Evaluates Large Vision-Language Models in realistic telemedicine consultations by simulating multi-agent dialogues between a doctor and a temperament-based patient. It probes diagnostic accuracy from multimodal inputs (images + text) and assesses clinical competence and dialogue quality.

Datasets

• 3MDBench — total ?; splits: train (-1), val (-1); repo https://github.com/univanxx/3mdbench

Metrics

• F1 Score (primary) — range: percent
  • Computed between diagnoses extracted by the LVLM and human-extracted ground truth diagnoses. Standard precision/recall F1 for diagnosis extraction.
• Cohen's Kappa — range: [0, 1]
  • Measures agreement between the Assessor Agent and human annotations across dialogue criteria. Averaged across criteria to determine overall agreement.

Input / output format

Input: Multimodal input per instance: patient medical image(s) + text dialogue history (complaints, symptoms, follow-ups). Doctor agent may also receive rationale or external diagnostic cues (e.g., top-3 ConvNet predictions).

Output: Text output containing extracted diagnosis(es) and clinical recommendations. Dialogue turns are generated iteratively by the Doctor Agent (max 512 tokens, temp 0.6).

Scoring recipe

pred_diagnoses = extract_diagnoses(model_output)
gold_diagnoses = human_ground_truth
tp = len(set(pred_diagnoses) & set(gold_diagnoses))
fp = len(set(pred_diagnoses) - set(gold_diagnoses))
fn = len(set(gold_diagnoses) - set(pred_diagnoses))
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
# Statistical testing
p_value = wilcoxon_signed_rank_test(metric_values_group1, metric_values_group2)
adjusted_p = benjamini_hochberg_fdr_control(p_values, alpha=0.01)

Common pitfalls

• Models often miss key symptoms or terminate questioning prematurely, unlike human clinicians who adaptively probe.
• Performance heavily depends on disease category (e.g., dermatology F1 ~47% vs throat/mucosae ~86%) and patient temperament (phlegmatic patients yield lower scores due to brevity).
• Rationale generation improves diagnostic F1 but can decrease clinical competence scores, indicating a trade-off between internal reasoning and applied communication skills.

Evidence (verbatim from paper)

For the diagnosis extraction task, we computed the F1 score of the diagnoses identified by the LVLM, using human-extracted diagnoses as the ground truth. To estimate the dialogue closeness, we calculated Cohen’s Kappa for each criterion from Table[2] and averaged the scores to determine overall agreement.

Citation

@misc{sviridov20253mdbench,
  title={3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark},
  author={Sviridov et al. (2025)},
  year={2025},
  note={arXiv:2504.13861}
}

• arXiv: 2504.13861