Intersection Scenarios Eval

intersection-scenarios-eval

A Reinforcement Learning Benchmark for Autonomous Driving in Intersection Scenarios — Liu et al. (2021) (arXiv:2109.10557, 2021)

What this evaluates

Evaluates reinforcement learning agents' ability to navigate complex, un-signalized urban intersections under varying traffic conditions. It probes decision-making, collision avoidance, and route completion in dynamic environments with interacting social vehicles.

Datasets

• Intersection Scenarios (RL-CIS) — total ?; splits: test (-1); repo https://github.com/liuyuqi123/ComplexUrbanScenarios

Metrics

• Success rate(%) (primary) — range: percent
  • Percentage of test episodes where the ego vehicle successfully completes the assigned route without collision or failure. Calculated as (successful runs / total runs) × 100.
• Average time(s) — range: other
  • Mean time in seconds taken by the agent to complete the route across all evaluated episodes.

Input / output format

Input: High-resolution CARLA simulator observations including ego-vehicle state, surrounding traffic flow parameters, and route waypoints.

Output: Continuous control actions (steering, acceleration, braking) or discrete maneuver commands per simulation step.

Scoring recipe

def compute_metrics(successes, total_runs, times):
    success_rate = (sum(successes) / total_runs) * 100
    avg_time = sum(times) / len(times)
    return success_rate, avg_time

Common pitfalls

• Deterministic vs. stochastic traffic flow generation significantly impacts results; stochastic uses uniform sampling while deterministic uses logical scenario definitions.
• Rule-based agents have limited input and cannot detect potential conflicts from cross directions, leading to lower safety and success rates compared to RL agents.
• Success rate and average time are calculated per functional scenario (turning left, turning right, going straight), not aggregated globally, so cross-task comparisons require careful normalization.

Evidence (verbatim from paper)

We evaluate the TD3 agent and rule-based agents in all five functional scenarios. Since the rule-based agents are poorly performed relatively. The statistics are calculated by the task routes for the rules-based agents. In turning left and turning right experiments, the RL agent reaches a success near $90%$ , and exceeds the rules-based agent in both success rate and average time.

Citation

@misc{liu2021reinforcement,
  title={A Reinforcement Learning Benchmark for Autonomous Driving in Intersection Scenarios},
  author={Liu et al. (2021)},
  year={2021},
  note={arXiv:2109.10557}
}

• arXiv: 2109.10557