Discharge Note Generation Eval

discharge-note-generation-eval

Enhancing Clinical Efficiency through LLM: Discharge Note Generation for Cardiac Patients — Jung et al. (2024) (arXiv:2404.05144, 2024)

What this evaluates

Evaluates the ability of fine-tuned LLMs to generate clinically accurate, complete, and readable discharge summaries for cardiac patients from raw medical records. It probes domain-specific medical summarization, factual consistency, and adherence to clinical documentation standards.

Datasets

• Cardiology Clinical Dataset — total ?; splits: test (-1)

Metrics

• ROUGE-1 / ROUGE-2 / ROUGE-L — range: [0, 1]
  • Overlap of unigrams, bigrams, and longest common subsequence between generated and reference text.
• BLEU — range: [0, 1]
  • Geometric mean of modified n-gram precisions with brevity penalty.
• BERTScore — range: [0, 1]
  • Cosine similarity between contextual embeddings of generated and reference tokens, aggregated via F1.
• Perplexity — range: [0, inf)
  • Exponential of the average negative log-likelihood of the reference text under the model.
• Accuracy (primary) — range: [1, 5]
  • Expert-rated score (1-5) for factual correctness of the generated discharge note, averaged across 5 test samples.

Input / output format

Input: Raw patient medical records / clinical notes from a cardiology center.

Output: Structured discharge note containing sections: [Chief Complaint], [Operation and Procedure], [Hospital Course], [Condition at Discharge], [Type of Discharge].

Scoring recipe

# Quantitative metrics (per sample)
rouge = compute_rouge(reference, generated)
bleu = compute_bleu(reference, generated)
bertscore = compute_bertscore(reference, generated)
ppl = compute_perplexity(reference, model)

# Qualitative expert assessment (5 test samples)
scores = []
for sample in test_samples[:5]:
    scores.append(expert_rating(sample.generated_note, criteria=[
        "Accuracy", "Completeness", "Readability & Comprehensibility",
        "Consistency", "Utility"
    ]))
total_qualitative = sum(scores)  # Max 25

Common pitfalls

• Expert qualitative assessment was only performed on 5 test samples, limiting statistical power and generalizability.
• Metrics like ROUGE and BERTScore are optimized for English and may not accurately reflect quality for Korean clinical notes without language-specific tokenization.
• Perplexity measures language model likelihood, not clinical factual accuracy or safety.

Evidence (verbatim from paper)

Five sample notes from the test set were evaluated by a cardiology expert across five criteria: Accuracy, Completeness, Readability and Comprehensibility, Consistency, and Utility, with each aspect rated on a 5-point scale.

Citation

@misc{jung2024discharge,
  title={Enhancing Clinical Efficiency through LLM: Discharge Note Generation for Cardiac Patients},
  author={Jung et al. (2024)},
  year={2024},
  note={arXiv:2404.05144}
}

• arXiv: 2404.05144