Medical Summarization Eval

medical-summarization-eval

Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settings — Balde et al. (2025) (arXiv:2505.21242, 2025)

What this evaluates

This evaluation probes the ability of large language models to generate accurate and faithful summaries of medical texts under high out-of-vocabulary (OOV) conditions. It specifically measures how tokenization fragmentation and domain-specific terminology affect summarization quality and concept preservation across multiple medical benchmarks.

Datasets

• PubMedQA — total 500; splits: test (500)
• EBM — total 424; splits: test (424)
• BioASQ-M — total 963; splits: test (963)
• BioASQ-S — total 496; splits: test (496)

Metrics

• Rouge-L (primary) — range: [0, 1]
  • Measures the longest common subsequence between the generated summary and the reference summary to assess informativeness and coherence.
• Concept-Score — range: [0, 1]
  • Measures faithfulness by computing the overlap of UMLS medical concepts extracted from both the generated and reference summaries using the QuickUMLS tool.

Input / output format

Input: Query appended to a PubMed abstract (or relevant snippets) as the source document, formatted into an in-context learning prompt with similarity-sampled examples.

Output: A concise text summary addressing the query based on the provided context.

Scoring recipe

def compute_rouge_l(predictions, references):
    scores = []
    for pred, ref in zip(predictions, references):
        scores.append(rouge_score(ref, pred, rouge_types=['rougeL']))
    return sum(scores) / len(scores)

def compute_concept_score(predictions, references):
    scores = []
    for pred, ref in zip(predictions, references):
        gen_concepts = quickumls.extract(pred)
        ref_concepts = quickumls.extract(ref)
        overlap = len(gen_concepts & ref_concepts)
        union = len(gen_concepts | ref_concepts)
        scores.append(overlap / union if union > 0 else 0.0)
    return sum(scores) / len(scores)

Common pitfalls

• High OOV concentration in medical texts causes severe tokenization fragmentation, which can artificially depress ROUGE scores independent of actual summarization quality.
• Concept-Score depends on the UMLS ontology via QuickUMLS, meaning it may fail to capture novel or non-standard medical terms not present in the reference ontology.
• In-context learning examples are selected using PubMedBERT similarity, which may not perfectly align with semantic relevance for medical summarization tasks.

Evidence (verbatim from paper)

We evaluate the model-generated summaries using Rouge-L (R-L) to measure informativeness and coherence, and Concept-Score (CSr) to measure faithfulnessZhang et al. ([2023]). Concept-Score measures the overlap of UMLS medical concepts (computed using QuickUMLS toolSoldaini and Goharian ([2016])) between the generated and reference summaries. We use Rouge-L as the primary comparison metric, in line with prior studiesFabbri et al. ([2021]); Yuan et al. ([2022]); Balde et al. ([2024a], [b]).

Citation

@misc{balde2025evaluation,
  title={Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settings},
  author={Balde et al. (2025)},
  year={2025},
  note={arXiv:2505.21242}
}

• arXiv: 2505.21242