# Wcep Mds Eval

> Evaluates multi-document summarization systems on news event clusters by measuring how well generated summaries match human-written reference summaries. It probes the model's ability to extract or generate concise, informative summaries from highly redundant, large-scale document collections. Use when the user wants to benchmark on WCEP, or asks about evaluating this task. Reports ROUGE F1-score.

- Skill: `qhjqhj00/wcep-mds-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/wcep-mds-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/wcep-mds-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/wcep-mds-eval

---


# wcep-mds-eval

> A Large-Scale Multi-Document Summarization Dataset from the Wikipedia Current Events Portal — Gholipour Ghalandari et al. (2020) (arXiv:2005.10070, 2020)

## What this evaluates

Evaluates multi-document summarization systems on news event clusters by measuring how well generated summaries match human-written reference summaries. It probes the model's ability to extract or generate concise, informative summaries from highly redundant, large-scale document collections.

## Datasets

- **WCEP** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/complementizer/wcep-mds-dataset

## Metrics

- `ROUGE F1-score` **(primary)** — range: [0, 1]
  - Computes n-gram overlap (1-gram, 2-gram, and longest common subsequence) between system and reference summaries, reporting F1-scores. Lowercased versions of summaries are used without further modification.

## Input / output format

**Input**: A cluster of news articles describing a single current event (truncated to a maximum of 100 articles per cluster in experiments).

**Output**: A summary of up to 40 tokens. Extractive methods return lists of full untruncated sentences; abstractive methods return generated text.

## Scoring recipe

```python
def compute_rouge_f1(predictions, references):
    preds = [p.lower() for p in predictions]
    refs = [r.lower() for r in references]
    scores = []
    for p, r in zip(preds, refs):
        r1 = rouge_1_f1(p, r)
        r2 = rouge_2_f1(p, r)
        rl = rouge_l_f1(p, r)
        scores.append((r1, r2, rl))
    return {
        'R1-F': sum(s[0] for s in scores) / len(scores),
        'R2-F': sum(s[1] for s in scores) / len(scores),
        'RL-F': sum(s[2] for s in scores) / len(scores)
    }
```

## Common pitfalls

- Summaries must be lowercased before scoring; otherwise ROUGE scores will be artificially low.
- The evaluation caps summary length at 40 tokens, but extractive methods must return full untruncated sentences that fit within this limit.
- The paper recommends evaluating with dynamic/longer output lengths and providing Recall results alongside the truncated F1 scores.

## Evidence (verbatim from paper)

> We evaluate lowercased versions of summaries and do not modify ground-truth or system summaries otherwise. We compare and evaluate systems using F1-score and Recall of ROUGE-1, ROUGE-2, and ROUGE-L (Lin, 2004).

## Citation

```bibtex
@misc{gholipourghalandari2020wcep,
  title={A Large-Scale Multi-Document Summarization Dataset from the Wikipedia Current Events Portal},
  author={Gholipour Ghalandari et al. (2020)},
  year={2020},
  note={arXiv:2005.10070}
}
```

- arXiv: 2005.10070