Sumo Highway Speed Eval

sumo-highway-speed-eval

Amortized Q-learning with Model-based Action Proposals for Autonomous Driving on Highways — Mirchevska et al. (2020) (arXiv:2012.03234, 2020)

What this evaluates

Evaluates the long-term driving consistency, trajectory quality, and lane-change efficiency of autonomous driving agents in varying traffic densities on simulated highways.

Datasets

• SUMO Highway Scenarios — total 80; splits: test (80)

Metrics

• average achieved speed (primary) — range: other
  • Mean vehicle speed over the evaluation horizon, averaged across 10 scenarios per traffic density level and 10 independent training runs per agent.

Input / output format

Input: RL state vector containing surrounding vehicle positions/velocities, gap features, and current lane information.

Output: Discrete option selection (keep lane, left/right lane-change) or continuous gap target for trajectory planning.

Scoring recipe

speeds = []
for run in range(10):
    for scenario in scenarios:
        speed = run_episode(agent, scenario)
        speeds.append(speed)
return sum(speeds) / len(speeds)

Common pitfalls

• Agents are evaluated on procedurally generated scenarios rather than a fixed public benchmark, making direct comparison with other papers difficult.
• The metric only reports average speed, ignoring safety violations or collision rates that may occur during lane changes.
• Hyperparameters were tuned via Hyperband over 500 runs, but final values are fixed per agent, potentially introducing selection bias.

Evidence (verbatim from paper)

We evaluated all agents on the same set of realistic scenarios, generated randomly, as described in [IV-E]. In order to objectively assess the driving of the agents, we generated scenarios with varying number of surrounding vehicles n={10,20,...,80}, simulating fluctuating traffic density, from light to heavy. For each number of surrounding vehicles we have 10 different evaluation scenarios (80 in total). We compared the average achieved speeds of all agents on the 80 scenarios. The average is calculated over the 10 scenarios with same number of cars in it. In order to acquire a better overview of the performance, for the random agent and the RL agents, we calculated an average over 10 different runs.

Citation

@misc{mirchevska2020amortized,
  title={Amortized Q-learning with Model-based Action Proposals for Autonomous Driving on Highways},
  author={Mirchevska et al. (2020)},
  year={2020},
  note={arXiv:2012.03234}
}

• arXiv: 2012.03234