Samsum Summarization Eval

samsum-summarization-eval

SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization — Gliwa et al. (2019) (arXiv:1911.12237, 2019)

What this evaluates

Abstractive summarization of naturalistic, messenger-style dialogues. It probes a model's ability to generate concise, human-like summaries from informal, multi-turn conversations containing typos, emoticons, and multi-party interactions.

Datasets

• SAMSum Corpus — total 16369; splits: train (-1), test (-1)

Metrics

• ROUGE F1 (ROUGE-1, ROUGE-2, ROUGE-L) (primary) — range: [0, 1]
  • Standard ROUGE metric reporting F1 scores with stemming for ROUGE-1, ROUGE-2, and ROUGE-L. F1 is the harmonic mean of recall and precision for n-gram overlaps.

Input / output format

Input: Dialogue conversation text (messenger-style, truncated to 400 tokens) with utterances separated by a special token (e.g., or |).

Output: Abstractive summary of the dialogue (no length limit during generation).

Scoring recipe

def compute_rouge_f1(predictions, references):
    # Uses py-rouge package with stemming enabled
    scores = {}
    for ngram in ['rouge1', 'rouge2', 'rougeL']:
        r = count_ngram_overlap(predictions, references, ngram, mode='recall')
        p = count_ngram_overlap(predictions, references, ngram, mode='precision')
        f1 = 2 * r * p / (r + p) if (r + p) > 0 else 0.0
        scores[ngram] = f1
    return scores

Common pitfalls

• ROUGE F1 scores on dialogue summarization often contradict human evaluator judgments, making them an imperfect proxy for summary quality.
• Models trained exclusively on news corpora (e.g., CNN/Daily Mail) fail to generalize to dialogue data without domain-specific training or joint training.
• The use of stemming in the ROUGE calculation differs from standard implementations, which can cause score discrepancies if not replicated exactly.

Evidence (verbatim from paper)

We evaluate models with the standard ROUGE metric (Lin, 2004), reporting the $F_{1}$ scores (with stemming) for ROUGE-1, ROUGE-2 and ROUGE-L following previous works (Chen and Bansal, 2018; See et al., 2017). We obtain scores using the py-roge package.

Citation

@misc{gliwa2019samsun,
  title={SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization},
  author={Gliwa et al. (2019)},
  year={2019},
  note={arXiv:1911.12237}
}

• arXiv: 1911.12237