Wcep Mds Eval

wcep-mds-eval

A Large-Scale Multi-Document Summarization Dataset from the Wikipedia Current Events Portal — Gholipour Ghalandari et al. (2020) (arXiv:2005.10070, 2020)

What this evaluates

Evaluates multi-document summarization systems on news event clusters by measuring how well generated summaries match human-written reference summaries. It probes the model's ability to extract or generate concise, informative summaries from highly redundant, large-scale document collections.

Datasets

• WCEP — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/complementizer/wcep-mds-dataset

Metrics

• ROUGE F1-score (primary) — range: [0, 1]
  • Computes n-gram overlap (1-gram, 2-gram, and longest common subsequence) between system and reference summaries, reporting F1-scores. Lowercased versions of summaries are used without further modification.

Input / output format

Input: A cluster of news articles describing a single current event (truncated to a maximum of 100 articles per cluster in experiments).

Output: A summary of up to 40 tokens. Extractive methods return lists of full untruncated sentences; abstractive methods return generated text.

Scoring recipe

def compute_rouge_f1(predictions, references):
    preds = [p.lower() for p in predictions]
    refs = [r.lower() for r in references]
    scores = []
    for p, r in zip(preds, refs):
        r1 = rouge_1_f1(p, r)
        r2 = rouge_2_f1(p, r)
        rl = rouge_l_f1(p, r)
        scores.append((r1, r2, rl))
    return {
        'R1-F': sum(s[0] for s in scores) / len(scores),
        'R2-F': sum(s[1] for s in scores) / len(scores),
        'RL-F': sum(s[2] for s in scores) / len(scores)
    }

Common pitfalls

• Summaries must be lowercased before scoring; otherwise ROUGE scores will be artificially low.
• The evaluation caps summary length at 40 tokens, but extractive methods must return full untruncated sentences that fit within this limit.
• The paper recommends evaluating with dynamic/longer output lengths and providing Recall results alongside the truncated F1 scores.

Evidence (verbatim from paper)

We evaluate lowercased versions of summaries and do not modify ground-truth or system summaries otherwise. We compare and evaluate systems using F1-score and Recall of ROUGE-1, ROUGE-2, and ROUGE-L (Lin, 2004).

Citation

@misc{gholipourghalandari2020wcep,
  title={A Large-Scale Multi-Document Summarization Dataset from the Wikipedia Current Events Portal},
  author={Gholipour Ghalandari et al. (2020)},
  year={2020},
  note={arXiv:2005.10070}
}

• arXiv: 2005.10070