Smd Few Shot Eval

smd-few-shot-eval

Data-Efficient Goal-Oriented Conversation with Dialogue Knowledge Transfer Networks — Shalyminov et al. (2019) (arXiv:1910.01302, 2019)

What this evaluates

Evaluates data efficiency and response generation quality in goal-oriented dialogue systems under few-shot conditions. It measures how well a model can generate contextually appropriate and entity-accurate responses using only a small fraction of in-domain dialogue data.

Datasets

• SMD — total ?; splits: test (-1)

Metrics

• BLEU (primary) — range: [0, 100] percent
  • Standard n-gram overlap score between predicted and gold responses, computed to conform with the Zhao and Eskenazi (2018) setup. It measures lexical similarity but does not account for semantic equivalence.
• Entity F1 — range: [0, 1]
  • F1 score measuring the precision and recall of correctly predicted entities in the response compared to the gold response. It evaluates goal-oriented accuracy rather than pure language generation.

Input / output format

Input: Dialogue context consisting of alternating user and system turns.

Output: A single predicted response string.

Scoring recipe

# Evaluate across multiple data percentages (e.g., 1%, 5%, 10%, 50%)
# Average over 10 runs with 10 different sets of seed dialogues per run
bleu_scores = []
entity_f1_scores = []
for run in range(10):
    for seed_set in range(10):
        pred = model.generate(context)
        bleu_scores.append(compute_bleu(gold, pred))
        entity_f1_scores.append(compute_entity_f1(gold, pred))
return mean(bleu_scores), mean(entity_f1_scores)

Common pitfalls

• High variance in reported results due to the evaluation protocol averaging over 10 runs with 10 different random seed dialogues.
• BLEU may penalize semantically correct responses that use different wording than the gold reference, which is common in open-ended dialogue generation.

Evidence (verbatim from paper)

We use BLEU as one of the main evaluation metrics in this paper — we do it in order to fully conform with the setup of Zhao and Eskenazi (2018) which we base our work on. But while being widely adopted as a general-purpose language generation metric, BLEU might not be sufficient in the dialogue setting (see Novikova et al. (2017) for a review). Specifically, we have observed several cases where the model would produce an overall grammatical response with the correct dialogue intent (e.g. "You are welcome! Anything else?"), but BLEU would output a lower score for it due to word mismatch (e.g. "You're welcome!"; see more examples in Table 4).

Citation

@misc{shalyminov2019dataefficient,
  title={Data-Efficient Goal-Oriented Conversation with Dialogue Knowledge Transfer Networks},
  author={Shalyminov et al. (2019)},
  year={2019},
  note={arXiv:1910.01302}
}

• arXiv: 1910.01302