qhjqhj00 pathology-vqa-eval
Exploring the Feasibility of Multimodal Chatbot AI as Copilot in Pathology Diagnostics: Generalist Model's Pitfall — Liu et al. (2024) (arXiv:2409.15291, 2024)
What this evaluates
Evaluates a multimodal chatbot's ability to interpret real-world pathology images (H&E and IHC) and integrate clinical context to produce accurate diagnoses, terminology, and multimodal reasoning across four anatomical systems.
Datasets
- Pathology Clinical Q&A Dataset — total 62; splits: test (62)
Metrics
diagnosis accuracy(primary) — range: [1, 5]- Expert-rated on a 1-5 scale (1=poor, 5=excellent); 0 indicates non-applicable. Final score per dimension is derived via consensus discussion among three senior pathologists.
terminology accuracy— range: [1, 5]- Expert-rated on a 1-5 scale (1=poor, 5=excellent); 0 indicates non-applicable. Final score per dimension is derived via consensus discussion among three senior pathologists.
multimodal information integration— range: [1, 5]- Expert-rated on a 1-5 scale (1=poor, 5=excellent); 0 indicates non-applicable. Final score per dimension is derived via consensus discussion among three senior pathologists.
diagnostic evidence accuracy on annotation— range: [1, 5]- Expert-rated on a 1-5 scale (1=poor, 5=excellent); 0 indicates non-applicable. Final score per dimension is derived via consensus discussion among three senior pathologists.
diagnostic evidence accuracy on texts— range: [1, 5]- Expert-rated on a 1-5 scale (1=poor, 5=excellent); 0 indicates non-applicable. Final score per dimension is derived via consensus discussion among three senior pathologists.
Input / output format
Input: H&E pathology image(s) (optionally with IHC images), patient clinical context (e.g., age, symptoms), and a diagnostic question.
Output: Textual response describing pathological features, differential diagnoses, and recommended immunohistochemical tests or final diagnosis.
Scoring recipe
def compute_metrics(responses, questions, images):
expert_scores = {dim: [] for dim in ['diagnosis_accuracy', 'terminology_accuracy', 'multimodal_integration', 'annotation_evidence', 'text_evidence']}
for resp, q, img in zip(responses, questions, images):
scores = [expert.score(resp, q, img) for expert in pathologists]
final = consensus_discussion(scores)
for dim in expert_scores:
expert_scores[dim].append(final[dim])
return {dim: mean(scores) for dim, scores in expert_scores.items()}
Common pitfalls
- Small sample size (62 rounds across 39 cases) limits statistical power and generalizability.
- Expert scoring relies on consensus discussion rather than independent aggregation, potentially introducing groupthink bias.
- Zero-score convention for 'non-applicable' cases is not averaged into the mean without clarification.
Evidence (verbatim from paper)
The responses from GPT were evaluated by three senior pathology experts. For each question answering, a final score was achieved after a joint review and discussion. The overall scores (Fig. 2A) indicate that ChatGPT is satisfactory in diagnostic annotation (average score of 3.64) and diagnosis accuracy (3.13) but shows room for improvement in terminology accuracy (2.38) and multimodal information integration (2.64).
Citation
@misc{liu2024multimodal,
title={Exploring the Feasibility of Multimodal Chatbot AI as Copilot in Pathology Diagnostics: Generalist Model's Pitfall},
author={Liu et al. (2024)},
year={2024},
note={arXiv:2409.15291}
}
- arXiv: 2409.15291