Usb Summarization Eval

usb-summarization-eval

USB: A Unified Summarization Benchmark Across Tasks and Domains — Kundan Krishna et al. (2023) (arXiv:2305.14296, 2023)

What this evaluates

Evaluates multiple text summarization capabilities including extractive/abstractive generation, factuality verification, factual error correction, topic-constrained generation, sentence compression, evidence extraction, and unsupported span detection across diverse domains.

Datasets

• Extractive Summarization (EXT) — total ?; splits: test (-1); repo https://github.com/kukrishna/usb
• Abstractive Summarization (ABS) — total ?; splits: test (-1); repo https://github.com/kukrishna/usb
• Factuality Classification (FAC) — total ?; splits: test (-1); repo https://github.com/kukrishna/usb
• Fixing Factuality (FIX) — total ?; splits: test (-1); repo https://github.com/kukrishna/usb
• Topic-based Summarization (TOPIC) — total ?; splits: test (-1); repo https://github.com/kukrishna/usb
• Multi-sentence Compression (COMP) — total ?; splits: test (-1); repo https://github.com/kukrishna/usb
• Evidence Extraction (EVEXT) — total ?; splits: test (-1); repo https://github.com/kukrishna/usb
• Unsupported Span Prediction (UNSUP) — total ?; splits: test (-1); repo https://github.com/kukrishna/usb

Metrics

• ROUGE (primary) — range: [0, 1]
  • Not specified in the provided section. Standard for summarization tasks (ABS, COMP, TOPIC). Typically computes n-gram overlap between generated and reference texts.

Input / output format

Input: Varies by task: (1) Full document; (2) Full document; (3) Pre/post-edited summary sentence + marked evidence; (4) Pre-edited summary sentence + marked evidence; (5) Full document + Wikipedia section header; (6) Cluster of source sentences; (7) Full source document + summary sentence; (8) Pre-edit summary sentence + marked evidence.

Output: Varies by task: (1) Set of source sentences; (2) Multi-sentence summary; (3) Factual/non-factual label; (4) Edited summary sentence; (5) Topic-specific summary sentence; (6) Single compressed sentence; (7) Minimal set of supporting source sentences; (8) Set of unsupported spans.

Scoring recipe

# Scoring recipe not specified in section.
# Standard protocol per subtask:
if task in [ABS, COMP, TOPIC]:
    score = rouge_l(predictions, gold)
elif task in [FAC]:
    score = accuracy(predictions, gold)
elif task in [FIX]:
    score = rouge_l(predictions, gold)
elif task in [EVEXT, UNSUP]:
    score = span_f1(predictions, gold)
elif task == EXT:
    score = set_f1(predictions, gold)
return score

Common pitfalls

• Ground truth references are task-specific (e.g., evidence sets, labels, or spans) rather than a single free-form summary.
• The benchmark emphasizes human-annotated data over heuristically generated data, which may affect generalization expectations.
• Multiple distinct tasks share similar inputs but require fundamentally different output formats (generation vs. classification vs. span prediction).

Evidence (verbatim from paper)

We derived labeled datasets for tasks using the collected annotations. The resulting benchmark consists of the following 8 tasks: Extractive Summarization (EXT): Given the full document as input, extract all important sentences that it contains. We define the ideal “reference” extractive summary as the set of all source sentences marked as evidence for the summary. Abstractive Summarization (ABS): Generate a multi-sentence summary... Factuality Classification (FAC): Predict if a summary sentence is factually correct... Fixing Factuality (FIX): Given a factually incorrect summary sentence, edit it... Topic-based Summarization (TOPIC): Given the source article and a topic... Multi-sentence Compression (COMP): Given a cluster of sentences... Evidence Extraction (EVEXT): Given a source document and a summary sentence, identify a minimal set of source sentences... Unsupported Span Prediction (UNSUP): Given a summary sentence and a set of sentences from the source providing evidence, predict spans in the summary which are not supported by the evidence.

Citation

@misc{krishna2023usb,
  title={USB: A Unified Summarization Benchmark Across Tasks and Domains},
  author={Kundan Krishna et al. (2023)},
  year={2023},
  note={arXiv:2305.14296}
}

• arXiv: 2305.14296