Summarization Fact Consistency Eval

summarization-fact-consistency-eval

Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency — Guo et al. (2022) (arXiv:2210.17378, 2022)

What this evaluates

Evaluates the factual consistency of human reference summaries across popular abstractive summarization benchmark datasets. It probes whether widely used datasets contain systematic factual errors or low-abstraction artifacts that compromise their validity as training and evaluation standards.

Datasets

• CNN/DM — total 311971; splits: full (311971)
• XSUM — total 226711; splits: full (226711)
• XL-Sum (English) — total 329592; splits: full (329592)

Metrics

• Factuality Score (primary) — range: [0, 1]
  • Computed using three off-the-shelf models (BERTScore_Art, BARTScore, DAE) that score the entailment or factual alignment of a reference summary given its source document. Higher scores indicate greater factual consistency.

Input / output format

Input: Source document and its corresponding human reference summary from benchmark datasets.

Output: A factual consistency score for each document-summary pair.

Scoring recipe

scores = []
for doc, summary in dataset:
    s_bert = BERTScore_Art(doc, summary)
    s_bart = BARTScore(doc, summary)
    s_dae = DAE(doc, summary)
    scores.append([s_bert, s_bart, s_dae])
# Aggregate per dataset to compare factuality levels
dataset_score = mean(scores, axis=0)

Common pitfalls

• Assuming human reference summaries are inherently factually correct 'gold standards'.
• Interpreting high factuality scores on CNN/DM as evidence of high abstraction quality, when they actually reflect low abstraction and bullet-point concatenation.
• Comparing datasets without accounting for differences in abstraction level or language coverage (e.g., XL-Sum covers 44 languages).

Evidence (verbatim from paper)

Here, we perform an analysis on the human references using the three factuality models introduced in Section 3.1: BERTScore_Art, BARTScore and DAE. The results are shown in Figure 3. We remark that factuality scores produced by all three models rank CNN/DM as the most factual dataset by a large margin.

Citation

@misc{guo2022questioning,
  title={Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency},
  author={Guo et al. (2022)},
  year={2022},
  note={arXiv:2210.17378}
}

• arXiv: 2210.17378