Tod Eval

tod-eval

Rewarding What Matters: Step-by-Step Reinforcement Learning for Task-Oriented Dialogue — Du et al. (2024) (arXiv:2406.14457, 2024)

What this evaluates

Evaluates a model's ability to perform multi-turn task-oriented dialogue by jointly tracking dialogue states and generating task-completing responses. It probes how well the system understands user intents, fills correct slots across multiple domains, and fulfills explicit user requests.

Datasets

• MultiWOZ2.0/2.1 — total 10438; splits: train (8438), val (1000), test (1000); HF multiwoz
• In-Car — total 3031; splits: train (2425), val (302), test (304)

Metrics

• Inform — range: [0, 1]
  • Measures whether the system provides correct entities based on its understanding of user requirements.
• Success — range: [0, 1]
  • Measures whether all user requests are met by the system.
• BLEU — range: [0, 100]
  • Standard n-gram overlap metric used to measure the fluency of the generated response.
• Comb (primary) — range: [0, 100]
  • Overall quality measure computed as (Inform + Success) × 0.5 + BLEU.
• Match — range: [0, 1]
  • Measures if a system can track all correct dialogue states to satisfy the user.
• SuccF1 — range: [0, 1]
  • Improves on Success by considering both recall (completeness) and precision (accuracy) to gauge response accuracy and completeness.

Input / output format

Input: Multi-turn dialogue history with delexicalized system responses and user utterances, including domain constraints and slot values.

Output: Delexicalized system response and/or predicted dialogue state (slot values).

Scoring recipe

def score_multiwoz(pred, gold):
    inform = 1.0 if set(pred.entities) == set(gold.entities) else 0.0
    success = 1.0 if all(req in pred.response for req in gold.requests) else 0.0
    bleu = compute_bleu(gold.response, pred.response)
    comb = (inform + success) * 0.5 + bleu
    return {'Inform': inform, 'Success': success, 'BLEU': bleu, 'Comb': comb}

def score_in_car(pred, gold):
    match = 1.0 if pred.states == gold.states else 0.0
    succ_f1 = f1_score(gold.requests, pred.requests)
    return {'Match': match, 'SuccF1': succ_f1}

Common pitfalls

• BLEU scores are often low because the model optimizes for task completion and understanding rather than linguistic fluency.
• Delexicalized responses require exact entity matching; mismatched entity IDs can artificially penalize Inform/Success scores.
• The Comb score masks trade-offs between understanding (Inform) and generation (BLEU), so reporting individual components is necessary.

Evidence (verbatim from paper)

For MultiWOZ, we report Inform and Success as introduced in [Section 4.1]. Additionally, we report BLEU that is used to measure the fluency of the generated response. Consequently, we report (Comb) that is computed by (Inform + Success) ×0.5 + BLEU as an overall quality measure. For In-Car, we leverage Match to measure if a system can track all correct states to satisfy the user. SuccF1 improves on the Success by considering both how completely (recall) and accurately (precision) the system handles requests.

Citation

@misc{du2024rewarding,
  title={Rewarding What Matters: Step-by-Step Reinforcement Learning for Task-Oriented Dialogue},
  author={Du et al. (2024)},
  year={2024},
  note={arXiv:2406.14457}
}

• arXiv: 2406.14457