Citesum Eval

citesum-eval

CiteSum: Citation Text-guided Scientific Extreme Summarization and Domain Adaptation with Limited Supervision — Mao et al. (2022) (arXiv:2205.06207, 2022)

What this evaluates

This benchmark evaluates a model's ability to generate concise, single-sentence summaries of scientific papers using citation sentences as ground truth. It probes extreme summarization capabilities and domain adaptation across academic disciplines.

Datasets

• CiteSum — total 92946; splits: train (83304), val (4721), test (4921); repo https://github.com/morningmoni/CiteSum

Metrics

• ROUGE-1, ROUGE-2, ROUGE-L (primary) — range: [0, 1]
  • Standard ROUGE recall metrics measuring n-gram (ROUGE-1/2) and longest common subsequence (ROUGE-L) overlap between the generated summary and the reference citation sentence.

Input / output format

Input: Abstract or full text of a scientific paper (average ~255 words)

Output: A single-sentence summary (citation sentence) of the cited paper (average ~23 words)

Scoring recipe

def compute_metrics(predictions, references):
    scores = {}
    for metric in ['ROUGE-1', 'ROUGE-2', 'ROUGE-L']:
        overlaps = [calculate_overlap(ref, pred) for ref, pred in zip(references, predictions)]
        scores[metric] = sum(overlaps) / len(overlaps)
    return scores

Common pitfalls

• Citation texts are not always accurate summaries; ~10% are misleading or partially accurate, so models may inherit noise.
• The dataset is heavily skewed toward computer science, mathematics, and engineering due to the Related Work section filtering, limiting generalizability to other disciplines.
• Splits must be paper-level to prevent label leakage, as a single paper can be cited multiple times with different citation sentences.

Evidence (verbatim from paper)

We set T to 50/20/40 for ROUGE-1/2/L recall through manual examination, resulting in a ROUGE-1/2/L recall of 73.1/39.4/58.5 after filtering. After data filtering and preprocessing, there are 92,946 examples in the final citation text-guided summarization dataset, which we name as CiteSum. We take about 5% of the data as the validation and test sets respectively, and the remaining 90% as the training set.

Citation

@misc{mao2022citesum,
  title={CiteSum: Citation Text-guided Scientific Extreme Summarization and Domain Adaptation with Limited Supervision},
  author={Mao et al. (2022)},
  year={2022},
  note={arXiv:2205.06207}
}

• arXiv: 2205.06207