Wikicatsum Rouge Eval

wikicatsum-rouge-eval

Generating Summaries with Topic Templates and Structured Convolutional Decoders — Perez-Beltrachini et al. (2019) (arXiv:1906.04687, 2019)

What this evaluates

Evaluates abstractive multi-document summarization models on their ability to generate coherent, content-adequate summaries across three domains (Company, Film, Animal). The protocol measures lexical and sentence-level overlap between generated summaries and reference summaries using ROUGE metrics, while also contextualizing scores against a baseline overlap between input documents and summaries.

Datasets

• WIKICATSUM — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• ROUGE-1, ROUGE-2, ROUGE-L (primary) — range: [0, 1]
  • ROUGE computes recall-based n-gram overlap (ROUGE-1 for unigrams, ROUGE-2 for bigrams) and longest common subsequence similarity (ROUGE-L) between the generated summary and the reference summary. Scores are reported as decimals between 0 and 1.

Input / output format

Input: First 800 tokens of multi-document input paragraphs concatenated with the Wikipedia lead section.

Output: Generated abstractive summary text.

Scoring recipe

def compute_rouge_metrics(generated_summary, reference_summary):
    import rouge
    scorer = rouge.Rouge()
    scores = scorer.get_scores(generated_summary, reference_summary, avg=True)
    return scores['rouge-1']['r'], scores['rouge-2']['r'], scores['rouge-l']['r']

Common pitfalls

• The paper computes a baseline ROUGE score between the input documents and summaries to contextualize model performance; this should not be confused with the standard model-to-reference evaluation.
• ROUGE scores are reported as decimals (e.g., .260) rather than percentages, which can cause misinterpretation when comparing to other literature using a 0-100 scale.
• Generation uses beam size 5 and trigram blocking (discarding steps with >80% token overlap), which directly impacts ROUGE scores and must be replicated for fair comparison.

Evidence (verbatim from paper)

We compute recall ROUGE scores of the input documents against the summaries to assess the amount of overlap and as a reference for the interpretation of the scores achieved by the models. Across domains content overlap (R1) is ~ 50 points. However, R2 is much lower indicating that there is abstraction, paraphrasing, and content selection in the summaries with respect to the input.

Citation

@misc{perezbeltrachini2019generatingsummariestopic,
  title={Generating Summaries with Topic Templates and Structured Convolutional Decoders},
  author={Perez-Beltrachini et al. (2019)},
  year={2019},
  note={arXiv:1906.04687}
}

• arXiv: 1906.04687