Crosswoz Dst Eval

crosswoz-dst-eval

An Empirical Study of Cross-Lingual Transferability in Generative Dialogue State Tracker — Lin et al. (2021) (arXiv:2101.11360, 2021)

What this evaluates

Evaluates the ability of generative dialogue state tracking models to accurately predict and maintain the complete set of user intents (domain-slot-value triples) across dialogue turns. It specifically probes cross-lingual and cross-ontology transfer capabilities by measuring how well models trained on one language or ontology generalize to another.

Datasets

• CrossWOZ-en — total ?; splits: test (-1)

Metrics

• Joint Goal Accuracy (primary) — range: [0, 1]
  • The proportion of dialogue turns where the predicted dialogue states match entirely to the ground truth dialogue states.
• Slot F1 — range: [0, 1]
  • The macro-averaged F1 score computed across all slots in each dialogue turn.

Input / output format

Input: Dialogue context and current user utterance.

Output: A set of domain-slot-value triples representing the current dialogue state.

Scoring recipe

def compute_metrics(predictions, golds):
    # predictions/golds: list of sets of (domain, slot, value) tuples per turn
    jga_correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    jga = jga_correct / len(predictions)
    
    slot_f1_scores = []
    for p, g in zip(predictions, golds):
        all_slots = {s for _, s, _ in g}
        slot_f1s = []
        for slot in all_slots:
            gold_vals = {v for d, s, v in g if s == slot}
            pred_vals = {v for d, s, v in p if s == slot}
            tp = len(gold_vals & pred_vals)
            fp = len(pred_vals - gold_vals)
            fn = len(gold_vals - pred_vals)
            prec = tp / (tp + fp) if (tp + fp) > 0 else 0
            rec = tp / (tp + fn) if (tp + fn) > 0 else 0
            f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
            slot_f1s.append(f1)
        slot_f1_scores.append(sum(slot_f1s) / len(slot_f1s) if slot_f1s else 0)
    slot_f1 = sum(slot_f1_scores) / len(slot_f1_scores)
    return jga, slot_f1

Common pitfalls

• Joint Goal Accuracy requires exact set equality per turn; partial slot matches do not contribute to the score.
• Slot F1 is macro-averaged across slots, meaning rare slots contribute equally to the final metric as frequent ones, which can mask performance on dominant slots.
• Cross-lingual transfer experiments often suffer from ontology misalignment, where identical slot names map to different semantic concepts across languages, unfairly penalizing the model.

Evidence (verbatim from paper)

We use joint goal accuracy and slot F1 as our metrics to evaluate our dialogue state tracking system.

• Joint Goal Accuracy: The proportion of dialogue turns where predicted dialogue states match entirely to the ground truth dialogue states.
• Slot F1: The macro-averaged F1 score for all slots in each turn.

Citation

@misc{lin2021crosslingual,
  title={An Empirical Study of Cross-Lingual Transferability in Generative Dialogue State Tracker},
  author={Lin et al. (2021)},
  year={2021},
  note={arXiv:2101.11360}
}

• arXiv: 2101.11360