Narrasum Eval

narrasum-eval

NarraSum: A Large-Scale Dataset for Abstractive Narrative Summarization — Zhao et al. (2022) (arXiv:2212.01476, 2022)

What this evaluates

This benchmark evaluates a model's ability to perform abstractive and extractive summarization on long-form narrative texts (movie/TV plot descriptions). It probes the model's capacity to capture event causality, character motivations, temporal dynamics, and overall narrative coherence while maintaining faithfulness to the source document.

Datasets

• NarraSum — total 122000; splits: train (-1), val (-1), test (-1); repo https://github.com/zhaochaocs/narrasum

Metrics

• ROUGE F1 (primary) — range: percent
  • F1 score computed at unigram (R-1), bigram (R-2), and longest common subsequence (R-L) overlap levels between generated and reference summaries.
• SummaC — range: [0, 1]
  • Automatic measure for summary faithfulness that detects inconsistencies between the summary and the source document, specifically designed for long inputs and outputs.
• Human Evaluation Scores — range: [1, 5]
  • Average score on a 5-point Likert scale rated by human annotators across four dimensions: Fluency, Faithfulness, Coherence, and Informativeness.

Input / output format

Input: Long-form narrative document (movie or TV episode plot description)

Output: Generated abstractive or extractive summary

Scoring recipe

# Compute ROUGE F1 scores
rouge1 = rouge(predictions, references, n=1).f1
rouge2 = rouge(predictions, references, n=2).f1
rougeL = rouge(predictions, references, lcs=True).f1

# Compute SummaC for abstractive models only
sc_scores = [summac_faithfulness(pred, gold) for pred, gold in zip(predictions, references)]

# Human evaluation averages (5-point Likert)
human_avg = {dim: np.mean(ratings[dim]) for dim in ["Fluency", "Faithfulness", "Coherence", "Informativeness"]}

Common pitfalls

• SummaC is explicitly noted as only applicable to abstractive summaries, as extractive ones are considered faithful by design.
• Human evaluation was restricted to the top 2 performing models (T5-Large, LED-Large) on a random sample of 100 test instances, not the full test set.
• Compression degree analysis requires splitting the test set by summary-to-document length ratio, which may skew results if not stratified properly.

Evidence (verbatim from paper)

We evaluate the generated summaries using ROUGE $F_{1}$ score. $^{8}$ We further include SummaC (Laban et al., 2022), an automatic measure for summary faithfulness. It achieves state-of-the-art on the benchmark of summary inconsistency detection, and is feasible to be applied to long input and output.

Citation

@misc{zhao2022narrasum,
  title={NarraSum: A Large-Scale Dataset for Abstractive Narrative Summarization},
  author={Zhao et al. (2022)},
  year={2022},
  note={arXiv:2212.01476}
}

• arXiv: 2212.01476