Scitldr Eval

scitldr-eval

TLDR: Extreme Summarization of Scientific Documents — Cachola et al. (2020) (arXiv:2004.15011, 2020)

What this evaluates

This benchmark evaluates the ability of models to generate extreme, single-sentence summaries (TLDRs) of scientific papers, capturing key contributions while bypassing background details. It tests both automated overlap metrics and human-judged informativeness and correctness under multi-target and multi-input settings.

Datasets

• SCITLDR — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/allenai/scitldr

Metrics

• Rouge-1 (primary) — range: percent
  • Unigram overlap between generated TLDR and gold TLDRs. Score per paper is the maximum Rouge-1 score across all available gold TLDRs (author-written and reviewer-derived).
• Rouge-2 — range: percent
  • Bigram overlap metric computed identically to Rouge-1, taking the maximum over all gold TLDRs per paper.
• Rouge-L — range: percent
  • Longest common subsequence overlap metric computed identically to Rouge-1, taking the maximum over all gold TLDRs per paper.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank of length-normalized nugget counts for predicted vs gold TLDRs, used for human informativeness evaluation.
• Correctness — range: [1, 3]
  • Mean author-assigned score on a 1-3 scale (1=false/misleading, 2=partially accurate, 3=mostly correct).

Input / output format

Input: Text of the paper's abstract, or the combined Abstract, Introduction, and Conclusion (AIC) sections.

Output: A single sentence TLDR summarizing the paper's key contributions.

Scoring recipe

def compute_rouge_max(predictions, gold_lists):
    paper_scores = []
    for pred, golds in zip(predictions, gold_lists):
        scores = [rouge_score(pred, g) for g in golds]
        paper_scores.append(max(scores))
    return sum(paper_scores) / len(paper_scores)

Common pitfalls

• Averaging scores across multiple gold TLDRs instead of taking the maximum, which penalizes models for matching any valid summary.
• Failing to length-normalize nugget counts when computing human informativeness, which biases longer outputs.
• Evaluating on full-text input when the benchmark explicitly defines abstract-only and AIC (Abstract+Intro+Conclusion) settings.

Evidence (verbatim from paper)

Following recent work on extreme summarization (Narayan et al., 2018; Lewis et al., 2020), we use Rouge-1, Rouge-2, and Rouge-L (Lin, 2004) as our automated metrics. As discussed in Section 2, we have multiple target summaries available per paper. To exploit this during evaluation, we calculate the Rouge score of the system-generated TLDR with respect to each of the gold TLDRs for the corresponding paper (including its TLDR-Auth and all of its TLDRs-PR) individually. We take the maximum Rouge score over these gold TLDRs as the final Rouge score for that paper.

Citation

@misc{cachola2020tldr,
  title={TLDR: Extreme Summarization of Scientific Documents},
  author={Cachola et al. (2020)},
  year={2020},
  note={arXiv:2004.15011}
}

• arXiv: 2004.15011