Diagnosisarena Eval

diagnosisarena-eval

DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models — Zhu et al. (2025) (arXiv:2505.14107, 2025)

What this evaluates

Clinical diagnostic reasoning capability of LLMs, requiring them to generate plausible diagnoses from patient case descriptions and imaging/symptom details. It probes the model's ability to perform complex, multi-step medical deduction and generalize across 28 clinical specialties.

Datasets

• DiagnosisArena — total 1113; splits: test (-1); repo https://github.com/SPIRAL-MED/DiagnosisArena

Metrics

• accuracy (primary) — range: percent
  • GPT-4o judges the model's open-ended diagnostic output against the ground truth, classifying it as identical, relevant, or irrelevant. Only identical counts as correct. Accuracy is the percentage of cases classified as identical.
• top-k accuracy — range: percent
  • Hit rate of the correct diagnosis within the top k predicted outcomes generated by the model in descending order of confidence.
• multi-choice accuracy — range: percent
  • Rule-based extraction of the model's selected option from a four-choice question, compared against the fixed ground truth option.

Input / output format

Input: Clinical case descriptions including patient symptoms, imaging findings, and incidental observations, presented via a unified prompt instructing the model to generate diagnostic outcomes.

Output: Open-ended: Five possible diagnostic outcomes in descending order of confidence. Multi-choice: A single selected option from four predefined choices.

Scoring recipe

def score_open_ended(predictions, gold):
    judge = GPT4oJudge()
    correct = 0
    for pred, g in zip(predictions, gold):
        if judge.classify(pred, g) == 'identical':
            correct += 1
    return correct / len(predictions)

def score_topk(predictions, gold, k=3):
    correct = 0
    for preds, g in zip(predictions, gold):
        if g in preds[:k]:
            correct += 1
    return correct / len(predictions)

def score_mc(predictions, gold):
    correct = 0
    for pred, g in zip(predictions, gold):
        if extract_option(pred) == g:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Multiple-choice formats significantly inflate performance by narrowing the problem space, allowing models to rely on superficial cues rather than full diagnostic reasoning.
• Data leakage from pretraining corpora (e.g., medical journals) can artificially boost scores if cases overlap with training data.
• Open-ended evaluation relies on an LLM-as-a-judge (GPT-4o), which may introduce bias or inconsistency compared to rule-based exact matching.

Evidence (verbatim from paper)

For open-ended questions, the ground truth is a clear diagnostic conclusion. In clinical scenarios, medical diagnoses are generally classified into three categories: “identical”, “relevant”, and “irrelevant”. Therefore, we use GPT-4o as the judge to evaluate the output results into these three categories. Among them, only when the model’s result is judged to be “identical” is it considered correct. Additionally, we instruct the LLM to generate the $k$ possible diagnostic outcomes in descending order of confidence, and then calculate the Top $k$ accuracy, which is the hit rate of the correct answer within the top $k$ predicted results.

Citation

@misc{zhu2025diagnosisarena,
  title={DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models},
  author={Zhu et al. (2025)},
  year={2025},
  note={arXiv:2505.14107}
}

• arXiv: 2505.14107