Multiwoz Dialogue Eval

multiwoz-dialogue-eval

Goal-Embedded Dual Hierarchical Model for Task-Oriented Dialogue Generation — Lai et al. (2019) (arXiv:1909.09220, 2019)

What this evaluates

Evaluates the quality, diversity, and goal adherence of task-oriented dialogue generation models. It measures how well a model generates natural, diverse responses while correctly incorporating specified dialogue goals and slot values.

Datasets

• MultiWOZ — total 10423; splits: train (8423), val (1000), test (1000)

Metrics

• BLEU-4 (primary) — range: percent
  • Standard BLEU score using 4-gram precision with a brevity penalty, computed against reference utterances.
• D-1, D-2, D-U — range: percent
  • Distinctiveness metrics: number of unique unigrams (D-1), bigrams (D-2), and full utterances (D-U) normalized by the total count of tokens/utterances in the generated text.
• Multi-label F1 — range: percent
  • Precision, recall, and F1-score calculated for extracted slot values (e.g., address, booking status) compared to reference goal targets.

Input / output format

Input: Delexicalized dialogue history with goal embeddings, capped at 22 turns and 36 tokens per sequence. Vocabulary size is 4258 including slots and special tokens.

Output: Generated utterance sequence produced via greedy decoding.

Scoring recipe

def compute_metrics(predictions, references):
    # BLEU-4
    bleu4 = nltk.translate.bleu_score.sentence_bleu(references, predictions, weights=(0.25, 0.25, 0.25, 0.25))
    # Diversity
    d1 = len(set(predictions)) / len(predictions)
    d2 = len(set(ngrams(predictions, 2))) / len(predictions)
    du = len(set(predictions)) / total_utterances
    # Multi-label F1 for goal slots
    pred_slots = extract_slots(predictions)
    gold_slots = extract_slots(references)
    tp = len(pred_slots & gold_slots)
    fp = len(pred_slots - gold_slots)
    fn = len(gold_slots - pred_slots)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return bleu4, d1, d2, du, precision, recall, f1

Common pitfalls

• Delexicalization strips entity names, so models cannot rely on memorizing specific values and must generate based on slot placeholders.
• Diversity metrics (D-1, D-2, D-U) are normalized by total token/utterance count, not vocabulary size, making them sensitive to sequence length.
• Goal focus is evaluated via multi-label slot extraction precision/recall rather than exact dialogue state tracking or intent classification.

Evidence (verbatim from paper)

We employ a number of automatic metrics as well as human evaluations to benchmark competing models on quality, diversity, and goal focus: Quality. BLEU (Papineni et al., 2002), as BLEU-4 by default, is a word-overlap measure against references and commonly used by dialogue generation works to evaluate quality (2015; 2016b; 2016a; 2017; 2018). Lower N-gram B1, B2, B3 are also reported. Diversity. D-1, D-2, D-U: The distinctiveness denotes the number of unique unigrams, bigrams, and utterances normalized by each total count (Li et al., 2016a; Xu et al., 2018). These metrics are commonly used to evaluate the dialogue diversity. Goal Focus. A set of slots such as address are extracted from reference dialogues as multi-label targets. Generated slots in model's output dialogues are the predictions. We use the multi-label precision, recall, and F1-score as surrogates to measure the goal focus and achievement.

Citation

@misc{lai2019goal,
  title={Goal-Embedded Dual Hierarchical Model for Task-Oriented Dialogue Generation},
  author={Lai et al. (2019)},
  year={2019},
  note={arXiv:1909.09220}
}

• arXiv: 1909.09220