Text Summarization Eval

text-summarization-eval

Text Summarization Using Large Language Models: A Comparative Study of MPT-7b-instruct, Falcon-7b-instruct, and OpenAI Chat-GPT Models — Basyal et al. (2023) (arXiv:2310.10449, 2023)

What this evaluates

Evaluates the abstractive text summarization capability of large language models by measuring how well they condense news articles into coherent, factually faithful, and linguistically natural summaries compared to human-written references.

Datasets

• CNN/Daily Mail 3.0.0 — total 300000; splits: test (-1)
• XSum — total ?; splits: test (-1)

Metrics

• BLEU Score — range: [0, 1]
  • Computes modified n-gram precision between the generated summary and reference summary, typically with a geometric mean across n=1..4 and a brevity penalty to discourage overly short outputs.
• ROUGE Score — range: [0, 1]
  • Measures recall-based n-gram overlap (ROUGE-N) and longest common subsequence (ROUGE-L) between the generated summary and reference summaries to evaluate content coverage.
• BERT Score (primary) — range: [0, 1]
  • Uses contextual embeddings from a pre-trained BERT model to compute token-level cosine similarity between the generated summary and reference summary, aggregating precision, recall, and F1 scores.

Input / output format

Input: Raw news article text (field 'article' for CNN/Daily Mail, 'document' for XSum) passed as a prompt instructing the model to generate a summary.

Output: A single generated abstractive summary text string.

Scoring recipe

def compute_metrics(predictions, references):
    bleu = compute_bleu(references, predictions)
    rouge = compute_rouge(references, predictions)  # ROUGE-1, ROUGE-L
    bert = compute_bert_score(references, predictions)  # F1 score
    return {'bleu': bleu, 'rouge': rouge, 'bert': bert}

Common pitfalls

• BLEU and ROUGE rely on exact n-gram overlap, often penalizing valid paraphrases or factually correct summaries that use different wording.
• BERT Score results vary significantly depending on the specific BERT checkpoint and version used for embedding generation, making cross-study comparisons difficult without strict version control.
• The paper does not specify the exact n-gram orders (e.g., ROUGE-1 vs ROUGE-L) or smoothing methods for BLEU, which can lead to inconsistent metric values across implementations.

Evidence (verbatim from paper)

To assess the quality and effectiveness of the generated summaries, we employed a set of widely accepted evaluation metrics: BLEU Score[[9]]: BLEU is a metric employed to assess the quality of machine translations. It operates by measuring the similarity between n-grams present in machine-translated sentences and those in human-translated sentences. ROUGE Score[[10], [12]]: The ROUGE Score assesses the overlap of n-grams (sequences of words) between the generated summary and reference summaries. It considers metrics such as ROUGE-N (unigrams, bigrams, etc.) and ROUGE-L (longest common subsequence) to evaluate content overlap. BERT Score[[11], [12]]: The BERT Score utilizes contextual embeddings from the BERT model to measure the similarity between the generated summary and reference summaries.

Citation

@misc{basyal2023textsummarization,
  title={Text Summarization Using Large Language Models: A Comparative Study of MPT-7b-instruct, Falcon-7b-instruct, and OpenAI Chat-GPT Models},
  author={Basyal et al. (2023)},
  year={2023},
  note={arXiv:2310.10449}
}

• arXiv: 2310.10449