Long Doc Rouge Eval

long-doc-rouge-eval

A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents — Cohan et al. (2018) (arXiv:1804.05685, 2018)

What this evaluates

Evaluates the quality of abstractive summaries for long scientific documents by measuring n-gram overlap between generated text and reference abstracts.

Datasets

• arXiv — total ?; splits: test (-1)
• PubMed — total ?; splits: test (-1)

Metrics

• ROUGE-1 (primary) — range: percent
  • Full-length F-1 score based on unigram overlap between generated summary and reference abstract. Tokens are lowercased and tokenized using spaCy.
• ROUGE-2 — range: percent
  • Full-length F-1 score based on bigram overlap, computed with the same lowercased spaCy tokenization.
• ROUGE-3 — range: percent
  • Full-length F-1 score based on trigram overlap, computed with the same lowercased spaCy tokenization.
• ROUGE-L — range: percent
  • Full-length F-1 score based on the longest common subsequence between generated and reference summaries, computed with the same lowercased spaCy tokenization.

Input / output format

Input: Long scientific documents (arXiv/PubMed papers), truncated to 2000 tokens per document, 500 tokens per section, max 4 sections.

Output: Abstractive summary (abstract), generated via beam search with beam size 4, max length 210 tokens.

Scoring recipe

def compute_rouge_f1(generated, reference):
    # 1. Lowercase both strings
    gen = generated.lower()
    ref = reference.lower()
    # 2. Tokenize using spaCy (sentence and word level)
    gen_tokens = [token.text for token in nlp(gen)]
    ref_tokens = [token.text for token in nlp(ref)]
    # 3. Compute full-length F-1 score for specified ROUGE variant
    score = rouge_score(ref, gen, use_stemmer=False)
    return score['rouge1'].fmeasure * 100  # Return as percent

Common pitfalls

• ROUGE tokenization differs across implementations; this paper explicitly uses spaCy for both sentence and word tokenization, unlike the standard NLTK-based ROUGE.
• The paper lowercases all tokens before evaluation, which can inflate scores compared to case-sensitive ROUGE variants.
• ROUGE scores are inherently biased towards extractive methods that copy salient sentences, making direct comparison with abstractive models challenging.

Evidence (verbatim from paper)

evaluation was done using the ROUGE automatic summarization evaluation metric (Lin, 2004) with full-length F-1 ROUGE scores. We lowercase all tokens and perform sentence and word tokenization using spaCy (Honnibal and Johnson, 2015).

Citation

@misc{cohan2018discourse,
  title={A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents},
  author={Cohan et al. (2018)},
  year={2018},
  note={arXiv:1804.05685}
}

• arXiv: 1804.05685