qhjqhj00 tracsum-eval
TracSum: A New Benchmark for Aspect-Based Summarization with Sentence-Level Traceability in Medical Domain — Chu et al. (2025) (arXiv:2508.13798, 2025)
What this evaluates
Evaluates a model's ability to generate aspect-specific summaries from clinical abstracts and accurately cite the supporting source sentences. It probes factual recall, conciseness, and traceability in a medical domain setting.
Datasets
- TracSum — total 3500; splits: test (3500); repo https://github.com/chubohao/TracSum
Metrics
Claim Recall(primary) — range: [0, 1]- Proportion of gold summary claims that are entailed by the model's generated summary, assessed via a claim decomposition model and entailment evaluator.
Citation Recall— range: [0, 1]- Proportion of gold cited sentence indices that are correctly retrieved by the model.
Claim Precision— range: [0, 1]- Proportion of claims in the model's summary that are entailed by the gold summary.
Citation Precision— range: [0, 1]- Proportion of model-cited sentence indices that match the gold citations.
Input / output format
Input: A clinical abstract represented as a sequence of uniquely indexed sentences, paired with a target medical aspect from a predefined set of seven (Aims, Intervention, Outcomes, Participants, Medicine, Duration, Side Effects).
Output: An aspect-specific summary string and a set of cited sentence indices from the input abstract. If no relevant information exists for the aspect, output 'Unknown' for the summary and 'Null' for citations.
Scoring recipe
def score(predictions, gold):
# predictions: (summary, cited_indices)
# gold: (gold_summary, gold_cited_indices)
claims_pred = decompose(predictions.summary)
claims_gold = decompose(gold.summary)
claim_recall = entailment_count(claims_pred, claims_gold) / len(claims_gold)
claim_precision = entailment_count(claims_gold, claims_pred) / len(claims_pred)
citation_recall = len(set(predictions.cited_indices) & set(gold.cited_indices)) / len(gold.cited_indices)
citation_precision = len(set(predictions.cited_indices) & set(gold.cited_indices)) / len(predictions.cited_indices)
return claim_recall, citation_recall, claim_precision, citation_precision
Common pitfalls
- Negative samples (where the abstract lacks information for the given aspect) must output 'Unknown' and 'Null' instead of hallucinating content.
- Citation metrics require exact sentence index matching, not just semantic similarity or paragraph-level grounding.
- Evaluation must be performed per aspect (7 instances per abstract) rather than aggregating across aspects, as each aspect targets different clinical information.
Evidence (verbatim from paper)
Completeness is assessed using Claim Recall and Citation Recall, while conciseness is measured by Claim Precision and Citation Precision. The system M(C', sum' | d, a) is expected to generate an aspect-specific summary sum' and a set of cited sentences C', where c'_i refers to the index of a sentence in d that supports the summary. If the article contains no information relevant to the given aspect, the system should output sum'←"Unknown" and C'←"Null".
Citation
@misc{chu2025tracsum,
title={TracSum: A New Benchmark for Aspect-Based Summarization with Sentence-Level Traceability in Medical Domain},
author={Chu et al. (2025)},
year={2025},
note={arXiv:2508.13798}
}
- arXiv: 2508.13798