qhjqhj00 multiwoz2.1-eval
Oh My Mistake!: Toward Realistic Dialogue State Tracking including Turnback Utterances — Kim et al. (2021) (arXiv:2108.12637, 2021)
What this evaluates
Evaluates a model's ability to track dialogue state (domain, slot, value triplets) across conversation turns, specifically probing its robustness to user mind-changes or 'turnback' utterances that modify previously stated intentions.
Datasets
- MultiWOZ 2.1 — total 10419; splits: train (8420), val (1000), test (999)
Metrics
joint goal accuracy(primary) — range: [0, 1]- For each turn, the model outputs a belief state (set of domain, slot, value triplets). The metric is marked as 1 if the predicted triplet set exactly matches the gold triplet set, and 0 otherwise. The final score is the average over all turns in the test set.
Input / output format
Input: Dialogue history consisting of alternating System and User utterances up to the current turn.
Output: Belief state represented as a set of (domain, slot, value) triplets.
Scoring recipe
correct = 0
total = 0
for turn in test_set:
pred_state = model.predict(turn.history)
gold_state = turn.gold_state
if set(pred_state) == set(gold_state):
correct += 1
total += 1
return correct / total
Common pitfalls
- Joint goal accuracy requires exact match of all slots in a turn; a single mismatch yields 0 for that turn.
- Performance drops on turnback scenarios stem from dataset bias (lack of turnback examples in training) rather than model architecture limitations.
- Template-generated turnbacks may overestimate model robustness compared to natural human turnbacks.
Evidence (verbatim from paper)
As a performance metric, the joint goal accuracy was employed. The joint goal accuracy is a standard criterion used to check if the model tracks the triplet of (domain, slot, value) precisely. When tracked correctly, the joint goal accuracy is marked as 1, and is otherwise 0. The numbers of training, validation, and test sets are 8420, 1000, and 999, respectively.
Citation
@misc{kim2021ohmymistake,
title={Oh My Mistake!: Toward Realistic Dialogue State Tracking including Turnback Utterances},
author={Kim et al. (2021)},
year={2021},
note={arXiv:2108.12637}
}
- arXiv: 2108.12637