Rl Dialogue Benchmark Eval

rl-dialogue-benchmark-eval

A Benchmarking Environment for Reinforcement Learning Based Task Oriented Dialogue Management — Casanueva et al. (2017) (arXiv:1711.11023, 2017)

What this evaluates

Evaluates the robustness and generalization of reinforcement learning-based dialogue management policies across varying simulated environments. It probes how well RL algorithms handle different domain sizes, user behavior profiles, and noisy speech input channels in task-oriented spoken dialogue systems.

Datasets

• PyDial simulated environments — total ?; splits: (unstated)

Metrics

• average success rate (primary) — range: [0, 1]
  • Percentage of dialogues completed successfully, i.e., whether the dialogue manager fulfills the user goal.
• average reward — range: other
  • 20 * 1(D) - T, where 1(D) is the success indicator and T is the dialogue length in turns.

Input / output format

Input: Dialogue state distribution factorized into ontology-defined slots and meta-data, processed through a rule-based dialogue state tracker, with noisy NLU inputs simulated via an error model.

Output: Summary actions from a predefined MDP action set (5 slot-independent + 3*|S| slot-dependent actions), subject to heuristic action masks based on the belief state.

Scoring recipe

def compute_metrics(dialogues):
    success_count = 0
    total_reward = 0.0
    for d in dialogues:
        goal_fulfilled = d.goal_achieved
        turns = len(d)
        if goal_fulfilled:
            success_count += 1
            total_reward += 20 - turns
        else:
            total_reward -= turns
    success_rate = success_count / len(dialogues)
    avg_reward = total_reward / len(dialogues)
    return success_rate, avg_reward

Common pitfalls

• The reward function penalizes every turn by -1, which may incentivize policies to terminate early or make suboptimal moves to minimize length rather than maximize goal fulfillment.
• Action masks are applied via heuristics dependent on the belief state, which can artificially restrict the effective action space and confound comparisons across different RL algorithms.
• Hyperparameters are strictly kept constant across all environments to test generalization, which may disadvantage algorithms that typically require domain-specific tuning.

Evidence (verbatim from paper)

The metrics presented in next section are the average success rate and average reward for each evaluated policy model. Success rate is defined as the percentage of dialogues which are completed successfully – i.e. whether the dialogue manager is able to fulfill the user goal or not. Final reward is defined as $20*\mathds{1}(\mathcal{D})-T$, where $\mathds{1}(\mathcal{D})$ is the success indicator and $T$ is the dialogue length in turns.

Citation

@misc{casanueva2017benchmarking,
  title={A Benchmarking Environment for Reinforcement Learning Based Task Oriented Dialogue Management},
  author={Casanueva et al. (2017)},
  year={2017},
  note={arXiv:1711.11023}
}

• arXiv: 1711.11023