Jama Clinical Challenge Eval

jama-clinical-challenge-eval

SemiHVision: Enhancing Medical Multimodal Models with a Semi-Human Annotated Dataset and Fine-Tuned Instruction Generation — Wang et al. (2024) (arXiv:2410.14948, 2024)

What this evaluates

Evaluates medical multimodal models on real-world diagnostic reasoning using clinical case images and questions. It probes both factual accuracy in close-ended QA and the model's ability to generate clinically sound reasoning across key points, inference steps, and evidence citation.

Datasets

• JAMA Clinical Challenge — total ?; splits: test (-1); repo https://github.com/believewhat/SemiHVision

Metrics

• Accuracy (primary) — range: [0, 1]
  • Standard close-ended QA accuracy calculated as the proportion of correct predictions over total instances.
• UMLS Factuality — range: [0, 1]
  • Measures factual consistency by computing the overlap between model outputs and gold answers using UMLS medical concept identifiers.
• GPT-4 Overall — range: other
  • Automated LLM-judge score evaluating the holistic quality and clinical appropriateness of the diagnostic response.
• GPT-4 Key-Points — range: other
  • LLM-judge score assessing the completeness of critical diagnostic points in the model's output.
• GPT-4 Inference — range: other
  • LLM-judge score measuring the logical reasoning steps and diagnostic deduction quality.
• GPT-4 Evidence — range: other
  • LLM-judge score evaluating the citation and relevance of supporting clinical evidence.

Input / output format

Input: Medical image(s) paired with a clinical question or case description requiring diagnosis or reasoning.

Output: Text response containing the diagnosis/answer, followed by reasoning steps, key points, inferences, and cited evidence.

Scoring recipe

def compute_metrics(predictions, golds, prompts):
    acc = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(predictions)
    umls_f = compute_uMLS_overlap(predictions, golds)
    gpt_scores = {dim: [] for dim in ['Overall', 'Key-Points', 'Inference', 'Evidence']}
    for p, g, prompt in zip(predictions, golds, prompts):
        judge_output = llm_judge(prompt, p, g, dimensions=['Overall', 'Key-Points', 'Inference', 'Evidence'])
        for dim in gpt_scores:
            gpt_scores[dim].append(parse_score(judge_output[dim]))
    return acc, umls_f, {k: mean(v) for k, v in gpt_scores.items()}

Common pitfalls

• The exact GPT-4 automated scoring prompts and rubrics are detailed in Section 4.4, which is not provided in this excerpt, making exact reproduction difficult.
• Traditional benchmarks (SLAKE, VQA-RAD) are heavily knowledge-recall biased (70-78% knowledge questions), whereas JAMA emphasizes inference (55% inference questions), so comparing raw accuracy across datasets is misleading.
• PathVQA dataset classification required expert intervention due to GPT-4o's domain knowledge gaps, highlighting potential bias in automated dataset analysis.

Evidence (verbatim from paper)

Accuracy was measured using standard methodologies for close-ended QA tasks, while diagnostic reasoning was evaluated through the automatic scoring pipeline described in Section 4.4, which measures performance across three key dimensions: Key Points, Inference, and Evidence.

Citation

@misc{wang2024semihvision,
  title={SemiHVision: Enhancing Medical Multimodal Models with a Semi-Human Annotated Dataset and Fine-Tuned Instruction Generation},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2410.14948}
}

• arXiv: 2410.14948