Summeval Eval

summeval-eval

SummEval: Re-evaluating Summarization Evaluation — Fabbri et al. (2020) (arXiv:2007.12626, 2020)

What this evaluates

This benchmark evaluates how well automatic summarization metrics align with human judgments across multiple quality dimensions. It probes whether standard n-gram, embedding-based, and reference-less metrics reliably predict human-perceived coherence, consistency, fluency, and relevance of generated summaries.

Datasets

• SummEval — total ?; splits: test (-1); repo https://github.com/Yale-LILY/SummEval

Metrics

• Kendall’s tau (primary) — range: [-1, 1]
  • A rank correlation coefficient measuring the correspondence between the rankings of automatic metric scores and human judgment scores across models or examples. Values range from -1 to 1, where 1 indicates perfect agreement.
• ROUGE-1 — range: [0, 1]
  • Measures unigram overlap between generated and reference summaries.
• BERTScore — range: [0, 1]
  • Computes token-level cosine similarity between contextualized BERT embeddings of candidate and reference texts.
• MoverScore — range: [0, 1]
  • Measures semantic distance using Word Mover’s Distance over n-gram embeddings pooled from BERT representations.

Input / output format

Input: Source document, reference summaries (for most metrics), and model-generated summary.

Output: Generated summary text.

Scoring recipe

def evaluate_summaries(generated_summaries, references, source_docs, human_judgments):
    auto_scores = {}
    for metric in ['ROUGE-1', 'BERTScore', 'MoverScore', 'BLEU', 'METEOR']:
        auto_scores[metric] = [compute_metric(metric, gen, refs) for gen, refs in zip(generated_summaries, references)]
    
    # System-level correlation with human judgments
    correlations = {}
    for metric, scores in auto_scores.items():
        tau = kendalltau(scores, human_judgments)
        correlations[metric] = tau
    return correlations

Common pitfalls

• ROUGE scores frequently correlate poorly with human judgments on fluency and coherence, leading to overestimation of model quality.
• Reference-less metrics like BLANC and SUPERT require the source document and cannot be computed without it.
• The benchmark emphasizes system-level correlation (Table 2) rather than instance-level accuracy, so per-example scores should not be treated as definitive quality indicators.

Evidence (verbatim from paper)

Table 2: Kendall’s tau correlation coefficients of expert annotations computed on a system-level along four quality dimensions with automatic metrics using 11 reference summaries per example.

Citation

@misc{fabbri2020summeval,
  title={SummEval: Re-evaluating Summarization Evaluation},
  author={Fabbri et al. (2020)},
  year={2020},
  note={arXiv:2007.12626}
}

• arXiv: 2007.12626