Tod Nlg Eval

tod-nlg-eval

Task-Oriented Dialogue System as Natural Language Generation — Wang et al. (2021) (arXiv:2108.13679, 2021)

What this evaluates

Evaluates the ability of task-oriented dialogue systems to generate natural language responses while maintaining entity consistency and completing user goals across multiple domains. It probes end-to-end dialogue generation, dialogue state tracking, and response quality under both automated simulation and human evaluation.

Datasets

• DSTC8 Track 1 End-to-End Multi-Domain Dialogue Challenge — total ?; splits: test (-1)
• MultiWOZ 2.0 benchmark — total ?; splits: test (-1)

Metrics

• Success Rate (primary) — range: [0, 1]
  • Dialogue is considered successful only if requestable slots are correctly filled and booking succeeds if needed. For MultiWOZ, it verifies whether all requested information has been answered.
• Book Rate — range: [0, 1]
  • Achieved only if the reserved information fits into all informable slots.
• Inform — range: [0, 1]
  • Measures whether a system has provided a correct entity.
• Success — range: [0, 1]
  • Verifies whether the system has answered all the requested information.
• BLEU — range: [0, 100]
  • Measures the fluency of the generated responses using standard n-gram overlap.
• Combined — range: [0, 1]
  • Computed as (Inform + Success) × 0.5 + BLEU to serve as an overall quality measure.
• Joint Accuracy — range: [0, 1]
  • Evaluates dialogue state tracking task accuracy.

Input / output format

Input: Dialogue history, user goals, and up to three database query results inserted between DST and POL modules.

Output: Natural language text representing system actions, belief states, and responses generated sequentially.

Scoring recipe

def compute_metrics(preds, gold, dataset):
    if dataset == 'dstc8':
        success = check_slots_filled(gold) and check_booking(gold)
        book_rate = check_reserved_info(gold)
        prec, rec, f1 = slot_filling_metrics(preds, gold)
        return success, book_rate, prec, rec, f1
    elif dataset == 'multiwoz':
        inform = check_entity_provided(preds, gold)
        success = check_all_requests_answered(preds, gold)
        bleu = compute_bleu(preds, gold)
        combined = (inform + success) * 0.5 + bleu
        joint_acc = check_dst_accuracy(preds, gold)
        return inform, success, bleu, combined, joint_acc

Common pitfalls

• Delexicalization preprocessing is explicitly removed in this setup, making entity generation harder and requiring direct copying mechanisms.
• Success Rate definition differs between DSTC8 (slot filling + booking) and MultiWOZ (answering all requested info).
• Human evaluation uses crowd-workers on Amazon Mechanical Turk rating on a 5-point scale, which may vary in annotator expertise.

Evidence (verbatim from paper)

The automatic and human evaluations of DSTC8 Track 1 is carried out by ConvLab... Automatic evaluation with user simulator: Success Rate, Book Rate, Return, Turns, Precision, Recall, F1. As for the Success Rate, the dialogue is considered as successful only if the requestable slots are correctly filled and book success if needed... We follow the automatic evaluation metrics to evaluate task completion and response quality for MultiWOZ 2.0 benchmark: Inform measures whether a system has provided a correct entity, Success verifies whether it has answered all the requested information, and BLEU is used to measure the fluency of the generated responses. A combined score (Combined) is also reported as an overall quality measure suggested in Mehri et al. ([2019]), which is computed with (Inform +Success)×0.5+BLEU.

Citation

@misc{wang2021taskoriented,
  title={Task-Oriented Dialogue System as Natural Language Generation},
  author={Wang et al. (2021)},
  year={2021},
  note={arXiv:2108.13679}
}

• arXiv: 2108.13679