Multiwoz Dst Eval

multiwoz-dst-eval

Dynamic Knowledge Fusion for Multi-Domain Dialogue State Tracking — Su et al. (2026) (arXiv:2603.10367, 2026)

What this evaluates

Evaluates a model's ability to track and predict dialogue states across multiple domains in a conversation. It measures how accurately the system maintains slot-value pairs as the user's goals evolve and switches between domains like restaurant, hotel, and taxi.

Datasets

• MultiWOZ — total 10000; splits: train|val|test (-1)

Metrics

• Joint Goal Accuracy (JGA) (primary) — range: [0, 1]
  • Proportion of dialogue turns where the predicted set of slot-value pairs exactly matches the ground truth set for that turn.
• Slot Accuracy (SA) — range: [0, 1]
  • Average accuracy across all slots in the ontology: SA = (Σ acc_i) / n, where n is the total number of ontology slots and acc_i is 1 if the predicted value matches the ground truth, else 0.

Input / output format

Input: Dialogue history (user and system utterances) up to the current turn, plus domain schema and ontology knowledge provided as contextual prompts.

Output: A set of slot-value pairs representing the predicted dialogue state for the current turn.

Scoring recipe

def compute_metrics(pred_states, gold_states, ontology):
    jga_correct = 0
    total_slots = len(ontology)
    total_slot_correct = 0
    for pred, gold in zip(pred_states, gold_states):
        if pred == gold:
            jga_correct += 1
        for slot in ontology:
            if pred.get(slot) == gold.get(slot):
                total_slot_correct += 1
    jga = jga_correct / len(pred_states)
    sa = total_slot_correct / (len(pred_states) * total_slots)
    return {'JGA': jga, 'SA': sa}

Common pitfalls

• JGA requires exact set matching of all slot-value pairs for a turn; partial matches do not count.
• Slot Accuracy is computed over the entire ontology size (n), not just the slots mentioned in the dialogue, which can dilute the score if many slots are irrelevant.
• Different MultiWOZ versions (2.1 vs 2.4) have significantly different annotation qualities and evaluation sets, making cross-version comparisons invalid without careful normalization.

Evidence (verbatim from paper)

Joint Goal Accuracy (JGA) quantifies the proportion of dialogue turns in which the model’s predicted dialogue state is an exact match to the label dialogue state, thus serving as a rigorous measure of full-state prediction accuracy. ... Slot Accuracy (SA) evaluates the correctness of the model’s predictions at the individual slot level for each dialogue turn. It is computed as the proportion of slot values that are predicted correctly, and is defined as: SA = \frac{\sum_{i}^{n}acc_{i}}{n}, where, n denotes the number of slots in the ontology.

Citation

@misc{su2026dynamic,
  title={Dynamic Knowledge Fusion for Multi-Domain Dialogue State Tracking},
  author={Su et al. (2026)},
  year={2026},
  note={arXiv:2603.10367}
}

• arXiv: 2603.10367