Schema Guided Dstc8 Eval

schema-guided-dstc8-eval

The SPPD System for Schema Guided Dialogue State Tracking Challenge — Li et al. (2020) (arXiv:2006.09035, 2020)

What this evaluates

Evaluates zero-shot dialogue state tracking across single and multi-domain conversations. It measures the model's ability to predict intents, extract slot values, and maintain accurate dialogue states over long contexts without prior exposure to unseen service domains.

Datasets

• Schema-Guided Dialogue (DSTC8 Track 4) — total ?; splits: train (-1), dev (-1)

Metrics

• Joint Goal Accuracy (primary) — range: percent
  • Exact match rate of all slot values across all domains in a dialogue turn.
• Average Goal Accuracy — range: percent
  • Mean of exact match rates computed per domain.
• Intent Accuracy — range: percent
  • Percentage of correctly predicted dialogue intents.
• Requested Slot F1 — range: percent
  • F1 score for correctly identifying requested slots.

Input / output format

Input: Lower-cased user utterances, system dialogue actions, and service schema definitions encoded via BERT.

Output: Predicted intent label, slot values for each domain, and requested slots.

Scoring recipe

def joint_goal_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred == gold:  # Exact match of all slot values across domains
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Evaluating only on seen domains ignores the zero-shot capability explicitly tested on unseen services.
• Using original system utterances instead of dialogue actions degrades performance, as the authors found actions yield better results.
• Joint goal accuracy requires exact match across all domains; partial matches or domain-wise averaging will overstate performance.

Evidence (verbatim from paper)

Table 2: The overall performance of our system

Citation

@misc{li2020sppd,
  title={The SPPD System for Schema Guided Dialogue State Tracking Challenge},
  author={Li et al. (2020)},
  year={2020},
  note={arXiv:2006.09035}
}

• arXiv: 2006.09035