Dialogue Summarization Eval

dialogue-summarization-eval

Unsupervised Abstractive Dialogue Summarization with Word Graphs and POV Conversion — Park et al. (2022) (arXiv:2205.13108, 2022)

What this evaluates

Evaluates the quality of unsupervised abstractive dialogue summarization across multiple domains by comparing generated summaries against human references using standard n-gram and LCS overlap metrics. It tests the model's ability to compress and rephrase conversational transcripts into coherent summaries without training data.

Datasets

• AMI — total ?; splits: test (-1)
• ICSI — total ?; splits: test (-1)
• DialogSum — total ?; splits: test (-1)
• SAMSum — total ?; splits: test (-1)
• MediaSum — total ?; splits: test (-1)
• SummScreen — total ?; splits: test (-1)
• ADS — total ?; splits: test (-1)

Metrics

• ROUGE-1 (primary) — range: percent
  • F-1 score based on unigram overlap between the generated summary and the reference summary.
• ROUGE-2 — range: percent
  • F-1 score based on bigram overlap between the generated summary and the reference summary.
• ROUGE-L — range: percent
  • F-1 score based on longest common subsequence (LCS) overlap between the generated summary and the reference summary.

Input / output format

Input: Dialogue transcript (optionally pre-segmented by topic for long texts >5,000 characters).

Output: A single abstractive summary sentence or paragraph.

Scoring recipe

def compute_rouge_f1(pred, ref, ngram_type):
    pred_ngrams = extract_ngrams(pred, ngram_type)
    ref_ngrams = extract_ngrams(ref, ngram_type)
    overlap = count_overlap(pred_ngrams, ref_ngrams)
    prec = overlap / len(pred_ngrams) if len(pred_ngrams) > 0 else 0
    rec = overlap / len(ref_ngrams) if len(ref_ngrams) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
# ngram_type: 'unigram' for ROUGE-1, 'bigram' for ROUGE-2, 'lcs' for ROUGE-L

Common pitfalls

• Topic segmentation is only applied to datasets with average transcription length greater than 5,000 characters (MediaSum, SummScreen), which changes the input structure for those splits.
• POV conversion is applied to all datasets to transform semi-extractive outputs into fully abstractive ones, meaning the raw graph path is not the final output.
• ROUGE scores are reported as percentages (e.g., 20.79), not decimals, which can cause confusion if standard 0-1 scaling is assumed.

Evidence (verbatim from paper)

We evaluate the quality of generated system summaries against reference summaries using standard ROUGE scores (Lin, 2004). Specifically, we use ROUGE-1 (R1), ROUGE-2 (R2), and ROUGE-L (RL) scores that respectively measure unigram, bigram, and longest common subsequence coverage.

Citation

@misc{park2022unsupervised,
  title={Unsupervised Abstractive Dialogue Summarization with Word Graphs and POV Conversion},
  author={Park et al. (2022)},
  year={2022},
  note={arXiv:2205.13108}
}

• arXiv: 2205.13108