Dst Jga Eval

dst-jga-eval

Contextual Semantic Parsing for Multilingual Task-Oriented Dialogues — Moradshahi et al. (2021) (arXiv:2111.02574, 2021)

What this evaluates

Evaluates a model's ability to track dialogue state in task-oriented conversations by predicting slot-value pairs across multiple domains. It measures how well the model maintains accurate belief states over multi-turn interactions, both with its own previous predictions and with ground-truth history.

Datasets

• RiSAWOZ — total 10000; splits: train (-1), val (-1), test (-1)
• MultiWOZ — total 8438; splits: train (-1), val (-1), test (-1)
• CrossWOZ — total 5012; splits: train (-1), val (-1), test (-1)

Metrics

• Joint Goal Accuracy (JGA) (primary) — range: percent
  • Exact match (EM) accuracy averaged across all turns in the test set. For each turn, the predicted belief state must exactly match the ground-truth belief state across all slots to count as correct.
• Gold Joint Goal Accuracy (GJGA) — range: percent
  • Same as JGA but uses the ground-truth belief state from previous turns as input instead of the model's predicted states, acting as an oracle to measure pure parsing accuracy without error compounding.

Input / output format

Input: Dialogue history (user and agent utterances), current user utterance, and either the model's predicted belief state from the previous turn (for JGA) or the ground-truth belief state (for GJGA).

Output: A belief state represented as a set of slot-value pairs for the current turn.

Scoring recipe

def compute_jga(predictions, golds):
    correct = 0
    for pred_state, gold_state in zip(predictions, golds):
        if pred_state == gold_state:  # Exact match across all slots
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Confusing JGA with GJGA: JGA uses the model's own predicted previous states, causing error compounding, while GJGA uses ground-truth previous states to isolate parsing performance.
• Not using exact match (EM) for all slots: The metric requires all slot assignments in a turn to match exactly; partial matches or fuzzy matching do not count.
• Ignoring dataset split conventions: MultiWOZ requires dropping hospital and police domains from training to match the validation and test splits used in prior work.

Evidence (verbatim from paper)

We evaluate the models using the following two metrics: Joint Goal Accuracy (JGA): The standard metric of evaluation in DST is joint goal accuracy, which measures the average accuracy of predicting all slot assignments to exact match (EM) for any given turn. To compute this metric for CSP, the belief state predicted in previous turn is used as input for the current turn. Gold Joint Goal Accuracy (GJGA): This metric is similar to JGA but is calculated on a turn by turn basis, with ground-truth belief state used as input. Assuming the belief state correctly captures the state up to current turn of the dialogue, this metric acts as an oracle in evaluation removing the compounding effect of errors from previous turns.

Citation

@misc{moradshahi2021contextual,
  title={Contextual Semantic Parsing for Multilingual Task-Oriented Dialogues},
  author={Moradshahi et al. (2021)},
  year={2021},
  note={arXiv:2111.02574}
}

• arXiv: 2111.02574