Medbench Eval

medbench-eval

MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models — Cai et al. (2023) (arXiv:2312.12806, 2023)

What this evaluates

This benchmark evaluates Chinese large language models on clinical knowledge, diagnostic reasoning, and conversational ability. It probes performance across three standardized medical licensing exams and real-world clinical case scenarios, highlighting gaps in multi-hop reasoning, diagnostic precision, and response fluency.

Datasets

• MedBench — total 41066; splits: test (41066)

Metrics

• accuracy (primary) — range: [0, 1]
  • Calculated as the number of correct predictions divided by the total number of questions. For multiple-choice items, a prediction is correct only if it exactly matches the ground-truth option letter.
• BLEU — range: [0, 1]
  • Standard n-gram precision metric (BLEU-1 and BLEU-4 reported) measuring lexical overlap between generated responses and reference clinical answers.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence metric measuring recall/precision of the longest matching word sequence between generated and reference responses.

Input / output format

Input: Multiple-choice medical exam questions (categorized as A1/A2/B or A3/A4 types) and real-world clinical case prompts covering Examinations, Treatments, and Diagnoses.

Output: For exam questions: a single letter choice (A-E). For clinical cases: a free-text generated response.

Scoring recipe

def compute_metrics(predictions, golds, case_preds, case_golds):
    # Exam accuracy
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    accuracy = correct / len(golds)
    
    # Clinical case BLEU & ROUGE-L
    bleu_scores = [compute_bleu(p, g) for p, g in zip(case_preds, case_golds)]
    rouge_scores = [compute_rouge_l(p, g) for p, g in zip(case_preds, case_golds)]
    return accuracy, bleu_scores, rouge_scores

Common pitfalls

• BLEU and ROUGE scores are inherently limited for clinical text and often yield moderate values even for top models, failing to holistically capture medical correctness or reasoning quality.
• Models frequently guess invalid options (e.g., 'F') on multiple-choice questions without proper prompting, artificially deflating accuracy if not filtered.
• Chain-of-Thought prompting yields inconsistent improvements across base models (e.g., helps Baichuan-13B but not ChatGLM-6B), so evaluation should account for prompt sensitivity.

Evidence (verbatim from paper)

In Table 1, we present a comprehensive analysis of the accuracy metrics for various LLMs across the three exams. We evaluated the outcomes using the BLEU and ROUGE F1-score metrics.

Citation

@misc{cai2023medbench,
  title={MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models},
  author={Cai et al. (2023)},
  year={2023},
  note={arXiv:2312.12806}
}

• arXiv: 2312.12806