Neural Medbench Eval

neural-medbench-eval

Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks — Miao Jing et al. (2025) (arXiv:2509.22258, 2025)

What this evaluates

Neural-MedBench probes the clinical reasoning and multimodal synthesis capabilities of vision-language models in neurology diagnostics. It specifically tests whether models can move beyond superficial classification to perform uncertainty resolution, generate clinically justified rationales, and maintain logical coherence when interpreting patient histories and medical imaging.

Datasets

• Neural-MedBench — total 200; splits: test (200)

Metrics

• Diagnostic Accuracy (pass@1) (primary) — range: [0, 1]
  • Binary metric indicating whether the model's top-1 predicted diagnosis exactly matches the gold-standard diagnosis.
• Diagnostic Accuracy (pass@5) — range: [0, 1]
  • Binary metric indicating whether the correct diagnosis appears anywhere within the model's top-5 differential diagnosis list.
• Semantic Fidelity (BERTScore) — range: [0, 1]
  • Computes the BERT-based F1 score between the model's generated rationale and the expert reference rationale to measure semantic alignment.
• Reasoning Fidelity (LLM Grader) — range: [0, 1]
  • A clinically calibrated GPT-4o model scores the output on correctness, logical coherence, and evidence grounding against clinical standards.

Input / output format

Input: A structured prompt initializing the model as an 'experienced neurologist', followed by a clinical task containing textual narratives, structured patient history, and medical imaging (MRI/CT encoded in base64).

Output: A differential diagnosis list (up to 5 options) and a free-form clinical rationale explaining the diagnostic reasoning.

Scoring recipe

def score(predictions, gold):
    # Diagnostic Accuracy
    pass1 = 1.0 if predictions['top1_diag'] == gold['diag'] else 0.0
    pass5 = 1.0 if gold['diag'] in predictions['top5_diag'] else 0.0
    
    # Semantic Fidelity
    bert_score = compute_bertscore(predictions['rationale'], gold['rationale'])
    
    # Reasoning Fidelity
    grader_prompt = build_clinical_grader_prompt(predictions['rationale'], gold['diag'])
    reasoning_score = gpt4o_grader.evaluate(grader_prompt) # Returns 0-1 or categorical
    
    return {'pass@1': pass1, 'pass@5': pass5, 'bertscore': bert_score, 'reasoning_fidelity': reasoning_score}

Common pitfalls

• Assuming high performance on broad classification benchmarks implies strong clinical reasoning competence, which the paper explicitly debunks.
• Relying solely on exact-match diagnosis accuracy without evaluating the quality of the generated rationale or differential diagnosis breadth.
• Failing to distinguish between perceptual errors (misreading images) and reasoning/knowledge gaps when analyzing model failures.

Evidence (verbatim from paper)

Diagnostic Accuracy (pass@k): For tasks with definitive outcomes, we report top-1 (pass@1) and top-5 (pass@5) accuracy. The latter measures whether the correct diagnosis appears within a model’s differential. Semantic Fidelity (BERTScore): Used for free-form rationale generation, capturing semantic alignment between model outputs and expert references. Reasoning Fidelity (LLM Grader): A clinically calibrated GPT-4o grader evaluates correctness, logical coherence, and evidence grounding (see Section[5.2]).

Citation

@misc{miao2025neuralmedbench,
  title={Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks},
  author={Miao Jing et al. (2025)},
  year={2025},
  note={arXiv:2509.22258}
}

• arXiv: 2509.22258