Sentence Summarization Eval

sentence-summarization-eval

Controlling the Amount of Verbatim Copying in Abstractive Summarization — Song et al. (2019) (arXiv:1911.10390, 2019)

What this evaluates

Evaluates abstractive summarization models on condensing source sentences into title-like summaries. It measures summary quality via lexical/semantic overlap and human judgments, while explicitly quantifying the degree of verbatim copying from the source text.

Datasets

• Gigaword — total 4011951; splits: train (4000000), val (10000), test (1951)
• Newsroom — total 241000; splits: train (199000), val (21000), test (21000)

Metrics

• ROUGE-2 (primary) — range: percent
  • Recall of overlapping 2-grams between the generated summary and the reference summary, expressed as a percentage.
• BERTScore — range: [0, 1]
  • Cosine similarity between BERT embeddings of generated and reference tokens, averaged across all tokens to measure semantic similarity.
• Copy Rate — range: percent
  • Percentage of summary n-grams (n=1,2,3,4) that appear verbatim in the source text, averaged across all n-gram orders.

Input / output format

Input: Source sentence (first sentence of a news article)

Output: Title-like summary (single sentence)

Scoring recipe

def compute_metrics(predictions, references, sources):
    rouge2_recall = rouge(predictions, references, rouge_types=['rouge2']).recall * 100
    bert_score_val = bert_score(predictions, references)
    copy_rates = []
    for pred, src in zip(predictions, sources):
        ngrams = [pred[i:i+n] for n in range(1,5) for i in range(len(pred)-n+1)]
        matches = sum(1 for ng in ngrams if ng in src)
        copy_rates.append(matches / len(ngrams) if ngrams else 0)
    avg_copy_rate = sum(copy_rates) / len(copy_rates) * 100
    return rouge2_recall, bert_score_val, avg_copy_rate

Common pitfalls

• Copy rate is averaged across 1-gram to 4-gram orders, not just unigrams.
• Human evaluation specifically checks grammaticality, informativeness, and factual consistency (true-to-original), not just overall fluency.
• Learning rate is halved when validation loss plateaus for 40,000 steps, which is critical for reproducing the reported results.

Evidence (verbatim from paper)

We evaluate our proposed method on the sentence summarization task. The goal is to condense a lengthy source sentence to a title-like summary. Comparing to single-document summarization, sentence summarization deals less with content selection; its ground-truth summaries also contain more paraphrasing and abstraction. We conduct experiments on the Gigaword (Parker 2011) and Newsroom (Grusky, Naaman, and Artzi 2018) datasets. Gigaword articles were collected during 1995-2010 and Newsroom spans the range of 1998-2017. We pair the first sentence of each article with its title to form an instance. The train-valid/test splits contain 4 million/10k/1951 instances for Gigaword and 199k/21k/21k instances for Newsroom. We experiment with both datasets to understand not only the copying behavior, but also domain adaptation effects for various models. Despite that only single reference summaries are available in benchmark evaluations, we are able to evaluate summary quality along multiple dimensions, using automatic metrics based on lexical similarity (ROUGE; Lin, 2004) and semantic similarity (BERTScore; Zhang et al., 2019), and through human assessment of grammaticality, informativeness, an

Citation

@misc{song2019controlling,
  title={Controlling the Amount of Verbatim Copying in Abstractive Summarization},
  author={Song et al. (2019)},
  year={2019},
  note={arXiv:1911.10390}
}

• arXiv: 1911.10390