Tempo Sum Eval

tempo-sum-eval

Can LMs Generalize to Future Data? An Empirical Analysis on Text Summarization — Cheang et al. (2023) (arXiv:2305.01951, 2023)

What this evaluates

Evaluates text summarization models' temporal generalization by testing on datasets split by publication date, specifically probing how well models handle knowledge-conflicting future articles versus in-distribution past data.

Datasets

• BBC — total ?; splits: in-distribution test (150), future test (150); repo https://github.com/NLP2CT/TempoSum
• CNN — total ?; splits: in-distribution test (150), future test (150); repo https://github.com/NLP2CT/TempoSum

Metrics

• FactCC (primary) — range: [0, 1]
  • Entailment-based faithfulness metric that checks if each claim in the generated summary is logically entailed by the source text using an NLI model.
• QAFactEval — range: [0, 1]
  • Uses question-generation and question-answering models to estimate summary faithfulness by verifying if answers to summary-derived questions align with the source text.

Input / output format

Input: Source news article text

Output: Generated summary text

Scoring recipe

def evaluate(predictions, sources):
    factcc_scores = []
    qafe_scores = []
    for pred, src in zip(predictions, sources):
        claims = extract_claims(pred)
        factcc_scores.append(mean([nli_entail(src, c) for c in claims]))
        questions = generate_questions(pred)
        answers = [qa_model(src, q) for q in questions]
        qafe_scores.append(estimate_faithfulness(answers, src))
    return {'FactCC': mean(factcc_scores), 'QAFactEval': mean(qafe_scores)}

Common pitfalls

• Models heavily rely on parametric world knowledge rather than the source text, leading to hallucinations of outdated facts on future data.
• Automatic metrics like FactCC and QAFactEval fail to reliably detect faithfulness improvements on future data, making human evaluation necessary for accurate assessment.

Evidence (verbatim from paper)

FactCC: An entailment-based faithfulness evaluation metric proposed by Kryscinski et al. (2020). Previous studies (Pagnoni et al., 2021) show that FactCC has a strong correlation with human judgments on existing benchmarks.

QAFactEval: The state-of-the-art faithfulness evaluation for text summarization (Fabbri et al., 2022). This metric utilizes question-generation and question-answering models to estimate the faithfulness of a summary.

Citation

@misc{cheang2023tempo-sum,
  title={Can LMs Generalize to Future Data? An Empirical Analysis on Text Summarization},
  author={Cheang et al. (2023)},
  year={2023},
  note={arXiv:2305.01951}
}

• arXiv: 2305.01951