Multiwoz 2.1 Eval

multiwoz-2.1-eval

MultiWOZ 2.1: A Consolidated Multi-Domain Dialogue Dataset with State Corrections and State Tracking Baselines — Budzianowski et al. (2019) (arXiv:1907.01669, 2019)

What this evaluates

Evaluates a model's ability to track and predict the complete set of user intent slots (dialogue state) across multiple domains in a multi-turn conversation.

Datasets

• MultiWOZ 2.1 — total ?; splits: train (-1), test (-1); repo https://github.com/budzianowski/multiwoz

Metrics

• slot accuracy (primary) — range: percent
  • The proportion of correctly predicted slot values across all slots in the dialogue. Calculated by dividing the total number of correctly predicted slot-value pairs by the total number of ground truth slot-value pairs.
• joint accuracy — range: percent
  • The proportion of turns where the model's predicted state exactly matches the ground truth state across all domains. Also referred to as turn-level accuracy in the text.

Input / output format

Input: Multi-turn dialogue history containing user utterances and system responses, segmented by domain.

Output: A dictionary mapping each domain to a set of slot-value pairs representing the predicted dialogue state.

Scoring recipe

def compute_metrics(predictions, gold):
    slot_correct = 0
    slot_total = 0
    joint_correct = 0
    for turn_pred, turn_gold in zip(predictions, gold):
        turn_slots_correct = 0
        for domain in turn_gold:
            for slot, value in turn_gold[domain].items():
                slot_total += 1
                if turn_pred[domain].get(slot) == value:
                    slot_correct += 1
                    turn_slots_correct += 1
        if turn_slots_correct == len(turn_gold):
            joint_correct += 1
    return slot_correct / slot_total, joint_correct / len(predictions)

Common pitfalls

• Models frequently misclassify the dontcare and none labels, which were intentionally increased in MultiWOZ 2.1 to better capture user ambiguity and penalize spurious predictions.
• Strict exact-match evaluation requires precise canonicalization of slot values; minor formatting or synonym differences result in hard penalties for both slot and joint accuracy.

Evidence (verbatim from paper)

For the Flat Joint State Tracker, we also observed that the largest slot accuracy decrease from MultiWOZ 2.0 to MultiWOZ 2.1 occurred for the restaurant.name slot (87.02%→ 83.33%).

Citation

@misc{budzianowski2019multiwoz,
  title={MultiWOZ 2.1: A Consolidated Multi-Domain Dialogue Dataset with State Corrections and State Tracking Baselines},
  author={Budzianowski et al. (2019)},
  year={2019},
  note={arXiv:1907.01669}
}

• arXiv: 1907.01669