Wikihow Eval

wikihow-eval

WikiHow: A Large Scale Text Summarization Dataset — Koupae et al. (2018) (arXiv:1810.09305, 2018)

What this evaluates

Evaluates text summarization systems on procedural, step-by-step articles written by non-journalists. It probes the model's ability to handle long sequences, non-inverted-pyramid structures, and high-abstraction content compared to standard news datasets.

Datasets

• WikiHow — total 230000; splits: (unstated)

Metrics

• ROUGE-L (primary) — range: [0, 1]
  • F1 score computed over the longest common subsequence of n-grams between the generated summary and the reference summary. Evaluated using exact matches, and optionally stem, paraphrase, and synonym matching (s/p/s).
• METEOR — range: [0, 1]
  • F1 score based on exact matches, stem matches, paraphrase matches, and synonym matches between the prediction and reference summary.

Input / output format

Input: Procedural article text containing step-by-step instructions.

Output: Generated summary text.

Scoring recipe

def evaluate(predictions, gold):
    # Uses Pyrouge package as specified in the paper
    rouge_scores = pyrouge.compute_rouge_scores(predictions, gold)
    meteor_scores = pyrouge.compute_meteor_scores(predictions, gold)
    # Returns F1 scores for ROUGE-1, ROUGE-2, ROUGE-L and METEOR
    return rouge_scores, meteor_scores

Common pitfalls

• Lead-3 baseline is adapted for WikiHow by concatenating the first sentence of each paragraph, rather than using the literal first three sentences of the article.
• Standard ROUGE/METEOR metrics may not fully capture the high abstraction level or compression ratio required for procedural text, often favoring extractive baselines.
• Sequence-to-sequence baselines without pointer-generator mechanisms fail on Out-Of-Vocabulary (OOV) words common in procedural articles.

Evidence (verbatim from paper)

To study the performance of the evaluated systems, we used the Pyrouge package to report the F1 score for ROUGE-1, ROUGE-2 and ROUGE-L (Lin, 2004) and the METEOR (Banerjee and Lavie, 2005) both based on the exact matches and on inclusion of stem, paraphrasing and synonyms $(s/p/s)$ to evaluate the methods.

Citation

@misc{koupae2018wikihow,
  title={WikiHow: A Large Scale Text Summarization Dataset},
  author={Koupae et al. (2018)},
  year={2018},
  note={arXiv:1810.09305}
}

• arXiv: 1810.09305