qhjqhj00 dst-eval
Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking — Moghe et al. (2021) (arXiv:2109.13620, 2021)
What this evaluates
Evaluates cross-lingual and zero-shot dialogue state tracking by measuring a model's ability to predict correct slot-value pairs in target languages using limited or translated training data.
Datasets
- Parallel MultiWoZ — total 10000; splits: train (-1), test (-1)
- Multilingual WoZ — total ?; splits: train (-1), test (-1)
Metrics
Joint Goal Accuracy(primary) — range: percent- Proportion of dialogue turns where the predicted dialogue state exactly matches the ground truth dialogue state.
Slot F1— range: percent- Macro-average of F1 scores computed over individual slot-types and slot-values for every turn.
Input / output format
Input: Current utterance concatenated with previous dialogue act, encoded by a multilingual transformer. For Parallel MultiWoZ, the state language is fixed (Chinese) while conversation language varies. For Multilingual WoZ, states are in the same language as the dialogue (German/Italian).
Output: Set of predicted slot-value pairs representing the dialogue state for the current turn.
Scoring recipe
def compute_jga(predictions, gold):
correct = sum(1 for pred, gt in zip(predictions, gold) if pred == gt)
return (correct / len(gold)) * 100
def compute_slot_f1(predictions, gold):
# Compute precision and recall for each slot-value type across all turns
# F1 = 2 * (precision * recall) / (precision + recall)
# Return macro-average F1 across all slot-value combinations
pass
Common pitfalls
- Slot Accuracy is explicitly discouraged because it treats unpredicted slots as correct, artificially inflating scores (e.g., 130 unpredicted out of 135 yields 96% accuracy despite errors).
- Cross-lingual evaluation setups differ between datasets: Parallel MultiWoZ uses language-agnostic states (Chinese) with mixed-language conversations, while Multilingual WoZ uses zero-shot language-matched states (German/Italian).
- Only 10% of target language training data is used for Parallel MultiWoZ, making low-resource performance the key focus rather than full supervised training.
Evidence (verbatim from paper)
The metrics used for dialogue state tracking tasks are turn-level and generally include Slot Accuracy, Slot F1, and Joint Goal Accuracy (JGA). Their descriptions are as follows:
Slot Accuracy: Proportion of the correct slots predicted across all utterances.
Slot F1: Macro-average of F1 score computed over the individual slot-types and slot-values for every turn.
Joint Goal Accuracy: Proportion of examples (dialogue turns) where the predicted dialogue state matches exactly the ground truth dialogue state.
Citation
@misc{moghe2021crosslingual,
title={Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking},
author={Moghe et al. (2021)},
year={2021},
note={arXiv:2109.13620}
}
- arXiv: 2109.13620