Multi News Eval

multi-news-eval

Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model — Fabbri et al. (2019) (arXiv:1906.01749, 2019)

What this evaluates

Evaluates abstractive models on multi-document summarization, measuring how well they condense multiple source articles into a single coherent summary. It probes coverage, redundancy control, and fluency under strict input length constraints.

Datasets

• Multi-News — total ?; splits: test (-1); repo https://github.com/Alex-Fabbri/Multi-News
• DUC 2004 — total ?; splits: test (-1)

Metrics

• R-1 (primary) — range: percent
  • Unigram overlap between generated summary and reference summary.
• R-2 (primary) — range: percent
  • Bigram overlap between generated summary and reference summary.
• R-SU (primary) — range: percent
  • Skip-gram overlap with a maximum gap of four words between tokens.
• Human Pairwise Wins — range: count
  • Count of times a system's summary was preferred over a baseline in pairwise comparisons across Informativeness, Fluency, and Non-Redundancy dimensions.

Input / output format

Input: Concatenated source documents truncated to exactly 500 tokens total. Tokens are distributed evenly (500/S per document), with a special token inserted between documents to mark boundaries.

Output: Abstractive summary text.

Scoring recipe

# ROUGE scoring
rouge_scores = {}
for metric in ['R-1', 'R-2', 'R-SU']:
    rouge_scores[metric] = rouge_score(generated, reference, metric=metric) * 100

# Human evaluation scoring
human_wins = {'Informativeness': 0, 'Fluency': 0, 'Non-Redundancy': 0}
for pair in pairwise_comparisons:
    for dim in ['Informativeness', 'Fluency', 'Non-Redundancy']:
        if pair.winner == 'system':
            human_wins[dim] += 1

Common pitfalls

• Truncation splits tokens evenly across documents (500/S) rather than taking full documents sequentially, which alters the input distribution.
• Cross-dataset evaluation trains on CNN/DailyMail and tests on DUC 2004, making results incomparable to in-domain Multi-News benchmarks.
• Human evaluation reports raw win counts rather than normalized scores or statistical significance, complicating direct metric comparison.

Evidence (verbatim from paper)

Following the setting from (Lebanoff et al., 2018), we report ROUGE (Lin, 2004) scores, which measure the overlap of unigrams (R-1), bigrams (R-2) and skip bigrams with a max distance of four words (R-SU). For the neural abstractive models, we truncate input articles to 500 tokens in the following way: for each example with S source input documents, we take the first 500 / S tokens from each source document.

Citation

@misc{fabbri2019multi_news,
  title={Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model},
  author={Fabbri et al. (2019)},
  year={2019},
  note={arXiv:1906.01749}
}

• arXiv: 1906.01749