# Discharge Note Generation Eval

> Evaluates the ability of fine-tuned LLMs to generate clinically accurate, complete, and readable discharge summaries for cardiac patients from raw medical records. It probes domain-specific medical summarization, factual consistency, and adherence to clinical documentation standards. Use when the user wants to benchmark on Cardiology Clinical Dataset, or asks about evaluating this task. Reports Accuracy.

- Skill: `qhjqhj00/discharge-note-generation-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/discharge-note-generation-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/discharge-note-generation-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/discharge-note-generation-eval

---


# discharge-note-generation-eval

> Enhancing Clinical Efficiency through LLM: Discharge Note Generation for Cardiac Patients — Jung et al. (2024) (arXiv:2404.05144, 2024)

## What this evaluates

Evaluates the ability of fine-tuned LLMs to generate clinically accurate, complete, and readable discharge summaries for cardiac patients from raw medical records. It probes domain-specific medical summarization, factual consistency, and adherence to clinical documentation standards.

## Datasets

- **Cardiology Clinical Dataset** — total ?; splits: test (-1)

## Metrics

- `ROUGE-1 / ROUGE-2 / ROUGE-L` — range: [0, 1]
  - Overlap of unigrams, bigrams, and longest common subsequence between generated and reference text.
- `BLEU` — range: [0, 1]
  - Geometric mean of modified n-gram precisions with brevity penalty.
- `BERTScore` — range: [0, 1]
  - Cosine similarity between contextual embeddings of generated and reference tokens, aggregated via F1.
- `Perplexity` — range: [0, inf)
  - Exponential of the average negative log-likelihood of the reference text under the model.
- `Accuracy` **(primary)** — range: [1, 5]
  - Expert-rated score (1-5) for factual correctness of the generated discharge note, averaged across 5 test samples.

## Input / output format

**Input**: Raw patient medical records / clinical notes from a cardiology center.

**Output**: Structured discharge note containing sections: [Chief Complaint], [Operation and Procedure], [Hospital Course], [Condition at Discharge], [Type of Discharge].

## Scoring recipe

```python
# Quantitative metrics (per sample)
rouge = compute_rouge(reference, generated)
bleu = compute_bleu(reference, generated)
bertscore = compute_bertscore(reference, generated)
ppl = compute_perplexity(reference, model)

# Qualitative expert assessment (5 test samples)
scores = []
for sample in test_samples[:5]:
    scores.append(expert_rating(sample.generated_note, criteria=[
        "Accuracy", "Completeness", "Readability & Comprehensibility",
        "Consistency", "Utility"
    ]))
total_qualitative = sum(scores)  # Max 25
```

## Common pitfalls

- Expert qualitative assessment was only performed on 5 test samples, limiting statistical power and generalizability.
- Metrics like ROUGE and BERTScore are optimized for English and may not accurately reflect quality for Korean clinical notes without language-specific tokenization.
- Perplexity measures language model likelihood, not clinical factual accuracy or safety.

## Evidence (verbatim from paper)

> Five sample notes from the test set were evaluated by a cardiology expert across five criteria: Accuracy, Completeness, Readability and Comprehensibility, Consistency, and Utility, with each aspect rated on a 5-point scale.

## Citation

```bibtex
@misc{jung2024discharge,
  title={Enhancing Clinical Efficiency through LLM: Discharge Note Generation for Cardiac Patients},
  author={Jung et al. (2024)},
  year={2024},
  note={arXiv:2404.05144}
}
```

- arXiv: 2404.05144