Driver Dojo Eval

driver-dojo-eval

Driver Dojo: A Benchmark for Generalizable Reinforcement Learning for Autonomous Driving — Rietsch et al. (2022) (arXiv:2207.11432, 2022)

What this evaluates

Evaluates the generalization capability of reinforcement learning policies for autonomous driving across procedurally generated traffic scenarios. It probes how well agents trained on a fixed set of road layouts and traffic dynamics perform when transferred to unseen environments with varying vehicle interactions and partial observability.

Datasets

• Driver Dojo — total ?; splits: train (-1), test (-1); repo https://github.com/seawee1/driver-dojo

Metrics

• Interquartile Mean (IQM) reward (primary) — range: other
  • The median performance of the middle 50% of evaluation runs. Calculated by sorting all run rewards and taking the mean of the central quartile.
• mean crash rate (CrR) — range: percent
  • The mean crash rate across evaluation runs, expressed as a percentage of episodes ending in a crash.
• mean completion rate (CoR) — range: percent
  • The mean completion rate across evaluation runs, expressed as a percentage of episodes successfully reaching the goal.

Input / output format

Input: Frame-stacked (5 frames) observations from feature-space observers (EgoState, TrafficState, RoadShape, Navigation) or image-based BirdEye outputs.

Output: Continuous control signals, discrete actions (5 equidistant values per control dimension), or semantic high-level commands.

Scoring recipe

def compute_metrics(rewards_per_run, crash_flags, completion_flags):
    sorted_rewards = sorted(rewards_per_run)
    n = len(sorted_rewards)
    lower = n // 4
    upper = 3 * n // 4
    iqm_reward = sum(sorted_rewards[lower:upper]) / (upper - lower)
    crash_rate = sum(crash_flags) / len(crash_flags) * 100
    completion_rate = sum(completion_flags) / len(completion_flags) * 100
    return iqm_reward, crash_rate, completion_rate

Common pitfalls

• Assuming that training on more diverse levels (e.g., 10K) always improves test performance; the paper finds 100 levels often outperform 10K due to training instability and network capacity limits.
• Evaluating on the same levels used for training, which fails to measure the benchmark's core goal of generalization to unseen scenarios.
• Ignoring stochasticity by reporting single-run results instead of aggregating across multiple seeds/runs to compute IQM.

Evidence (verbatim from paper)

Performance is quantified as the Interquartile Mean (IQM) reward, which is the median performance of the middle 50% of runs and promises to be more robust to outliers while being more statically efficient than median performance. We additionally report the mean crash rate (CrR) and mean completion rate (CoR) of evaluation runs. To test for generalizability, we train each algorithm on a fixed number of levels and evaluate on different set of levels.

Citation

@misc{rietsch2022driverdojo,
  title={Driver Dojo: A Benchmark for Generalizable Reinforcement Learning for Autonomous Driving},
  author={Rietsch et al. (2022)},
  year={2022},
  note={arXiv:2207.11432}
}

• arXiv: 2207.11432