Multiwoz Eval

multiwoz-eval

Domain State Tracking for a Simplified Dialogue System — Jeon et al. (2021) (arXiv:2103.06648, 2021)

What this evaluates

Evaluates end-to-end task-oriented dialogue systems on their ability to track user goals, fulfill multi-domain requests, and generate contextually appropriate responses. It specifically probes how well models maintain conversation state and achieve user objectives without relying on full historical dialogue context.

Datasets

• MultiWOZ 2.1 — total 10000; splits: train (8000), val (1000), test (1000)

Metrics

• inform rate (primary) — range: percent
  • Percentage of correctly provided goal attributes across all dialogues divided by the total number of goal attributes. The inform rate indicates how many entries provided by the system meet the goals.
• success rate (primary) — range: percent
  • Percentage of dialogues where the system successfully provides all requested attributes for every domain. The success rate additionally measures how many user requests are provided by the system.
• BLEU score — range: percent
  • Standard n-gram overlap metric comparing generated system responses against reference responses. The BLEU score measures the similarity between the generated response and the true response in the datasets.

Input / output format

Input: Current user utterance, updated domain state, and belief state (compact representation replacing full dialogue history).

Output: Natural language system response.

Scoring recipe

inform_count = 0
total_goals = 0
success_count = 0
for dialogue in dataset:
    total_goals += len(dialogue.goals)
    inform_count += count_correctly_provided_attributes(dialogue.system_outputs, dialogue.goals)
    if all_requested_attributes_provided(dialogue.system_outputs, dialogue.requests):
        success_count += 1
inform_rate = (inform_count / total_goals) * 100
success_rate = (success_count / len(dataset)) * 100
bleu = compute_bleu(dataset.references, dataset.predictions)

Common pitfalls

• Inform rate is calculated at the attribute level across all dialogues, while success rate is calculated at the dialogue level; confusing the two leads to incorrect reporting.
• BLEU scores in MultiWOZ are inherently low due to high paraphrasing variance in natural dialogue; relying solely on BLEU misrepresents task performance.
• Evaluating on MultiWOZ 2.0 without correcting known annotation errors can yield artificially inflated or deflated metrics compared to the standardized 2.1 split.

Evidence (verbatim from paper)

The evaluation metrics for end-to-end systems are the inform rate, success rate, and BLEU score. The inform rate indicates how many entries provided by the system meet the goals. The success rate additionally measures how many user requests are provided by the system. The BLEU score measures the similarity between the generated response and the true response in the datasets.

Citation

@misc{jeon2021domain,
  title={Domain State Tracking for a Simplified Dialogue System},
  author={Jeon et al. (2021)},
  year={2021},
  note={arXiv:2103.06648}
}

• arXiv: 2103.06648