Wikiasp Eval

wikiasp-eval

WikiAsp: A Dataset for Multi-domain Aspect-based Summarization — Hayashi et al. (2020) (arXiv:2011.07832, 2020)

What this evaluates

Evaluates multi-domain aspect-based summarization, requiring models to first discover relevant aspects (Wikipedia section titles) from cited references and then generate domain-specific summaries. It probes content selection, cross-document pronoun resolution, and temporal ordering in multi-source generation.

Datasets

• WikiAsp — total ?; splits: train (-1), test (-1); repo http://github.com/neulab/wikiasp

Metrics

• R-2 (primary) — range: [0, 1]
  • ROUGE-2 recall/precision/F1 based on bigram overlap between the generated summary and the reference summary.
• R-L — range: [0, 1]
  • ROUGE-L longest common subsequence overlap between generated and reference summaries.
• F-1 — range: [0, 1]
  • Harmonic mean of precision and recall for aspect classification.

Input / output format

Input: Cited reference texts and a target aspect (section title) for summarization; Wikipedia section and cited references for aspect discovery.

Output: Aspect label (classification) or generated summary text (summarization).

Scoring recipe

def score_aspect(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if p == g and p == 'aspect')
    fp = sum(1 for p, g in zip(preds, gold) if p == 'aspect' and g != 'aspect')
    fn = sum(1 for p, g in zip(preds, gold) if p != 'aspect' and g == 'aspect')
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def score_summ(gold, pred):
    return rouge_score(gold, pred, rouge_types=['rouge2', 'rougeL'])

Common pitfalls

• Class imbalance in aspect frequency distributions causes poorly calibrated classifiers that achieve high recall but low precision.
• Low ROUGE scores occur even with oracle baselines because important summary phrases are not rare, making n-gram overlap a poor proxy for content quality.

Evidence (verbatim from paper)

The automatic evaluation results are shown in Table 5. Neither baseline unanimously outperformed the other on all domains, but we observe that Pre-Summ (abstractive) performs better than TextRank (extractive) on average. The low R-2 and R-L scores by both models despite the oracle being relatively higher suggest that important phrases to be summarized do not appear rarely.[9] ... We show the aspect discovery results in Table 4. ... Class imbalance also plays a role here; predicting the major classes give high recall due to skew aspect frequency distributions.

Citation

@misc{hayashi2020wikiasp,
  title={WikiAsp: A Dataset for Multi-domain Aspect-based Summarization},
  author={Hayashi et al. (2020)},
  year={2020},
  note={arXiv:2011.07832}
}

• arXiv: 2011.07832