Multiwoz Dialog Summarization Eval

multiwoz-dialog-summarization-eval

Abstractive Dialog Summarization with Semantic Scaffolds — Lin Yuan et al. (2019) (arXiv:1910.00825, 2019)

What this evaluates

Evaluates abstractive summarization models on their ability to preserve critical semantic slots, entities, and domain consistency across multi-domain dialog conversations.

Datasets

• MultiWOZ — total ?; splits: test (-1)

Metrics

• ROUGE (primary) — range: [0, 1]
  • Standard ROUGE-L recall/precision/f1 comparing generated summaries to reference summaries.
• CIC — range: [0, 1]
  • Critical Information Coverage; measures the preservation of critical slot information and entities in the generated summary.
• Relevance — range: [1, 5]
  • Human-assigned score (1-5) indicating how well the summary captures the dialog's core information.
• Conciseness — range: [1, 5]
  • Human-assigned score (1-5) indicating the summary's brevity and lack of redundancy.
• Readability — range: [1, 5]
  • Human-assigned score (1-5) indicating the naturalness and fluency of the summary.
• Ranking Win/Lose/Tie — range: percent
  • Pairwise comparison percentage where workers rank one summary over another (or tie).

Input / output format

Input: Multi-domain dialog conversations with speaker-labeled utterances (e.g., Client/Help desk).

Output: Abstractive summary text.

Scoring recipe

# Automatic metrics
rouge = compute_rouge_l(predictions, references)
cic = compute_cic(predictions, references)  # Measures critical slot/entity preservation

# Human evaluation (100 test samples, 3 workers each)
human_scores = []
for sample in test_set:
    for worker in workers:
        rel, conc, read = worker.score(sample.summary, scale=1-5)
        human_scores.append({'relevance': rel, 'conciseness': conc, 'readability': read})
        rank = worker.rank_pair(sample.summary_a, sample.summary_b)  # Returns Win/Lose/Tie

Common pitfalls

• ROUGE scores can be high while CIC is low, meaning standard metrics miss critical slot/entity preservation.
• Ground truth references may omit necessary information or sound unnatural, limiting their reliability as a gold standard.
• Semantic slot scaffolds require explicit annotations that are rarely available in standard dialog corpora.

Evidence (verbatim from paper)

We observe that SPNet reaches the highest score in both ROUGE and CIC. Both Pointer-Generator and Transformer achieve high ROUGE scores, but a relative low CIC scores. It suggests that the baselines have more room for improvement on preserving critical slot information.

Citation

@misc{yuan2019abstractive,
  title={Abstractive Dialog Summarization with Semantic Scaffolds},
  author={Lin Yuan et al. (2019)},
  year={2019},
  note={arXiv:1910.00825}
}

• arXiv: 1910.00825