Multiwoz 2.1 Jga Eval

multiwoz-2.1-jga-eval

Improving Longer-range Dialogue State Tracking — Ye Zhang et al. (arXiv:2103.00109, 2021)

What this evaluates

Evaluates a model's ability to perform Dialogue State Tracking (DST) by predicting the correct values and statuses for all requested slots across multi-domain conversations. It specifically probes robustness to long-range contextual noise and class imbalance in slot status prediction.

Datasets

• MultiWOZ 2.1 — total ?; splits: test (7372)

Metrics

• joint-goal-accuracy (JGA) (primary) — range: [0, 1]
  • Ratio of turns where all dialogue states (slot values and statuses) are correctly predicted to the total number of turns.

Input / output format

Input: Multi-turn conversation history containing user and agent utterances across multiple domains.

Output: Dialogue state representation specifying the status (active/inactive) and value for each slot in the current turn.

Scoring recipe

correct_turns = 0
total_turns = 0
for turn in test_set:
    total_turns += 1
    pred_states = model.predict(turn.history)
    if pred_states == turn.gold_states:
        correct_turns += 1
jga = correct_turns / total_turns

Common pitfalls

• JGA is computed at the turn level, so a single incorrect slot in a turn causes the entire turn to be marked wrong.
• Slot status prediction suffers from severe class imbalance (most slots are inactive), making high accuracy easy to achieve trivially without meaningful prediction.

Evidence (verbatim from paper)

We use the commonly used joint-goal-accuracy (JGA) as metric for DST, defined as the ratio between turns whose all states are correctly predicted and total number of turns.

Citation

@misc{zhang2021improving,
  title={Improving Longer-range Dialogue State Tracking},
  author={Ye Zhang et al.},
  year={2021},
  note={arXiv:2103.00109}
}

• arXiv: 2103.00109