Booksum Eval

booksum-eval

BookSum: A Collection of Datasets for Long-form Narrative Summarization — Kryściński et al. (2021) (arXiv:2105.08209, 2021)

What this evaluates

Evaluates extractive and abstractive summarization models on long-form narrative texts across paragraph, chapter, and book granularities. It probes lexical overlap, semantic similarity, content coverage via question answering, and human-rated fluency, coherence, relevance, and factuality.

Datasets

• BookSum — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/salesforce/booksum

Metrics

• ROUGE-1 (primary) — range: [0, 100]
  • F1 score of unigram overlap between generated and reference summaries.
• BERTScore — range: [-1, 1]
  • Semantic overlap computed by aligning summaries at the token level using cosine similarity between contextual embeddings.
• SummaQA — range: [0, 1]
  • Content overlap measured by generating questions from the source document and evaluating how many can be answered correctly using the generated summary.
• Human Fluency/Coherence/Relevance/Factuality — range: [1, 5]
  • Average Likert scale ratings (1-5) assigned by three human annotators per dimension.

Input / output format

Input: Source narrative text (paragraph, chapter, or full book).

Output: Generated summary text (extractive sentences or abstractive text).

Scoring recipe

def score(preds, refs, sources):
    # ROUGE
    r1, r2, rl = rouge.compute(preds, refs, types=['rouge1','rouge2','rougeL'])
    # BERTScore
    bs = bertscore.compute(preds, refs, lang='en')
    # SummaQA
    qs = qa_gen.generate(sources)
    ans = qa_sys.predict(qs, summary=preds)
    sqa = mean(ans)
    # Human
    h = avg(annotators.score(preds, refs, sources, dims=['fluency','coherence','rel','fact']))
    return {'R-1': r1, 'R-2': r2, 'R-L': rl, 'BS': bs, 'SQA': sqa, 'Human': h}

Common pitfalls

• BERTScore yields consistently low scores for long abstractive summaries, suggesting it may be ill-suited for this domain.
• SummaQA and human relevance/factuality scores are limited by input length; chunking/aggregation or paragraph-only evaluation may reduce reliability.
• The generate-and-rank approach for chapters/books can produce summaries lacking coherence or containing overlapping/insignificant content.

Evidence (verbatim from paper)

Models were evaluated using a suite of automatic evaluation metrics included in the SummEval toolkit. Lexical overlap between n-grams in generated and reference summaries was measured using ROUGE-{1,2,L} metrics. Semantic overlap between mentioned summaries was evaluated using BERTScore, which aligns summaries on a token-level based on cosine similarity scores between token embeddings. We also inspect content overlap between generated summaries and source documents by employing SummaQA, which generates questions based on the input document and next applies a QA system to evaluate how many of those question can be answered using the summary.

Citation

@misc{krysinski2021booksum,
  title={BookSum: A Collection of Datasets for Long-form Narrative Summarization},
  author={Kryściński et al. (2021)},
  year={2021},
  note={arXiv:2105.08209}
}

• arXiv: 2105.08209