Xsum Extreme Summarization Eval

xsum-extreme-summarization-eval

Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization — Narayan et al. (2018) (arXiv:1808.08745, 2018)

What this evaluates

Evaluates a model's ability to perform extreme abstractive summarization by generating a single-sentence summary from a full news article, requiring synthesis, paraphrasing, and inference across document sections.

Datasets

• XSum — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/shashiongithub/XSum

Metrics

• automatic metrics (primary) — range: percent
  • Computes overlap scores (typically ROUGE-1, ROUGE-2, ROUGE-L) between the generated summary and the reference summary. Human evaluation assesses quality via crowdworkers. Exact formulas are not specified in this section.

Input / output format

Input: Lower-cased news article truncated to 400 tokens, optionally conditioned with a topic distribution derived from an LDA model.

Output: A single-line summary limited to 90 tokens.

Scoring recipe

# Generation
preds = beam_search(model, input_doc, beam_size=10)
# Evaluation
auto_score = compute_rouge(preds, gold_summary)
human_score = crowdworker_rating(preds, gold_summary)
return auto_score, human_score

Common pitfalls

• Input documents are strictly truncated to 400 tokens, which may remove salient information needed for summarization.
• Summaries are hard-limited to 90 tokens, potentially forcing compression of key details.
• Models are trained on lower-cased text, which may affect entity recognition and capitalization in outputs.
• Topic distribution is estimated via an external LDA model, adding preprocessing overhead and potential noise.

Evidence (verbatim from paper)

We report results with various systems which were all trained on the XSum dataset to generate a one-line summary given an input news article. ... During train ing and at test time the input document was truncated to 400 tokens and the length of the summary limited to 90 tokens. ... Summaries at test time were obtained using beam search (with beam size 10). ... outperforms extractive baselines and state-of-the-art RNN-based abstractive systems in both automatic metrics and human evaluation.

Citation

@misc{narayan2018dont,
  title={Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization},
  author={Narayan et al. (2018)},
  year={2018},
  note={arXiv:1808.08745}
}

• arXiv: 1808.08745