Task Oriented Dialogue Eval

task-oriented-dialogue-eval

A Network-based End-to-End Trainable Task-oriented Dialogue System — Wen et al. (2016) (arXiv:1604.04562, 2016)

What this evaluates

Evaluates the ability of neural dialogue systems to track user intent and belief states, generate contextually appropriate responses, and successfully complete task-oriented conversations (specifically restaurant search) by jointly modeling intent, belief, and database interaction.

Datasets

• Wizard-of-Oz restaurant dialogue corpus — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Objective task success rate (primary) — range: percent
  • Percentage of dialogues marked successful if both (1) the offered entity matches the user's request, and (2) the system answered all associated information requests (e.g., address, phone) from the user.
• BLEU score — range: [0, 1]
  • Standard n-gram overlap metric (Papineni et al., 2002) computed on template-like output sentences before lexicalization with entity value substitution. Reported for top-1 and top-5 candidates.
• Entity matching rate — range: percent
  • Percentage of dialogues where the actual selected entity at the end of the conversation matches the task specified to the user.
• Tracker F-1 score — range: percent
  • F-1 score computed from Precision and Recall for Informable and Requestable slot tracking.
• Human subjective success rate — range: percent
  • Percentage of dialogues rated as successful by human judges on Amazon Mechanical Turk.
• Comprehension score — range: 1 to 5
  • Human-rated comprehension ability on a 1 to 5 scale.
• Naturalness score — range: 1 to 5
  • Human-rated naturalness of response on a 1 to 5 scale.

Input / output format

Input: Delexicalized user utterances and dialogue history.

Output: Template-like system responses (before lexicalization with entity value substitution).

Scoring recipe

def compute_metrics(predictions, golds):
    bleu_scores = [bleu_score(gold, pred) for pred, gold in zip(predictions, golds)]
    match_rate = 0
    success_count = 0
    for pred, gold in zip(predictions, golds):
        entity_match = (pred.selected_entity == gold.specified_entity)
        requests_answered = all(gold.request in pred.answered_info for gold.request in gold.requests)
        if entity_match:
            match_rate += 1
        if entity_match and requests_answered:
            success_count += 1
    return {
        'BLEU': sum(bleu_scores) / len(bleu_scores),
        'Entity Matching Rate': match_rate / len(predictions),
        'Task Success Rate': success_count / len(predictions)
    }

Common pitfalls

• BLEU is computed on delexicalized/template-like outputs before entity substitution, not on fully lexicalized text.
• Task success requires BOTH correct entity matching AND answering all requested information; matching the entity alone is insufficient.
• Tracker metrics are split into Informable (e.g., food, area, pricerange) and Requestable (e.g., address, phone) slots, which should not be averaged without distinction.

Evidence (verbatim from paper)

Three evaluation metrics were used: BLEU score (on top-1 and top-5 candidates) (Papineni et al., 2002), entity matching rate and objective task success rate (Su et al., 2015). We calculated the entity matching rate by determining whether the actual selected entity at the end of each dialogue matches the task that was specified to the user. The dialogue is then marked as successful if both (1) the offered entity matches, and (2) the system answered all the associated information requests (e.g. what is the address?) from the user.

Citation

@misc{wen2016network,
  title={A Network-based End-to-End Trainable Task-oriented Dialogue System},
  author={Wen et al. (2016)},
  year={2016},
  note={arXiv:1604.04562}
}

• arXiv: 1604.04562