Clinical Note Eval

clinical-note-eval

Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes — Kweon et al. (2023) (arXiv:2309.00237, 2023)

What this evaluates

This evaluation probes the clinical reasoning, safety, and instruction-following capabilities of large language models on real-world medical datasets. It measures how well models generate accurate and appropriate responses to clinical prompts compared to baseline systems and GPT-3.5-turbo.

Datasets

• MIMIC-III — total 413; splits: test (413)
• MIMIC-IV — total 500; splits: test (500)
• i2b2 — total 619; splits: test (619)
• MTSamples — total 101; splits: test (101)
• CASI (AE) — total 100; splits: test (100)
• CASI (CR) — total 100; splits: test (100)
• DisCQ — total 100; splits: test (100)

Metrics

• scores (primary) — range: [1, 5]
  • Average rating assigned by evaluators (GPT-4 and/or clinicians) on a 1–5 scale for each generated response. Higher scores indicate better clinical accuracy, safety, and helpfulness.

Input / output format

Input: Clinical instruction or case description paired with a reference answer (gold), used to prompt the model for a response.

Output: A natural language clinical response or answer generated by the model.

Scoring recipe

scores = []
for pred, gold in zip(predictions, golds):
    rating = evaluator.evaluate(pred, gold)  # 1-5 scale
    scores.append(rating)
return sum(scores) / len(scores)

Common pitfalls

• Small test set sizes for several datasets (100–101 samples), which may lead to high variance in average scores.
• Reliance on LLM-as-a-judge (GPT-4) and limited human clinicians introduces potential bias and inconsistency in scoring criteria.
• Scores are reported per model size (7B, 13B, 175B) but not aggregated across sizes, making cross-architecture comparison difficult.

Evidence (verbatim from paper)

Table 2: Individual Experimental Results. Bolded entries denote the best scores, while underscored ones represent the second-best results for each model size.

Citation

@misc{kweon2023asclepius,
  title={Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes},
  author={Kweon et al. (2023)},
  year={2023},
  note={arXiv:2309.00237}
}

• arXiv: 2309.00237