Carla Real Traffic Scenarios Eval

carla-real-traffic-scenarios-eval

CARLA Real Traffic Scenarios -- novel training ground and benchmark for autonomous driving — Osiński et al. (2020) (arXiv:2012.11329, 2020)

What this evaluates

Evaluates the ability of reinforcement learning policies to execute tactical driving maneuvers (e.g., lane changes, roundabout navigation) in a simulated environment mapped from real-world traffic data. It probes how observation modalities and reward structures impact policy generalization and success rates across diverse driving scenarios.

Datasets

• NGSIM — total ?; splits: test (-1); repo https://github.com/deepsense-ai/carla-real-traffic-scenarios
• openDD — total ?; splits: test (-1); repo https://github.com/deepsense-ai/carla-real-traffic-scenarios

Metrics

• success rate (primary) — range: [0, 1]
  • The proportion of test scenarios where the policy successfully completes the target maneuver without crashing or violating constraints.

Input / output format

Input: Per instance: a tactical driving scenario (lane change or roundabout) with observations provided as either a bird’s-eye view tensor (186×150×5), visual camera feed, or LiDAR & camera data.

Output: Per instance: a sequence of continuous driving actions (steering, acceleration, braking) generated by the RL policy to navigate the scenario.

Scoring recipe

def compute_success_rate(predictions, gold):
    successful = 0
    total = len(predictions)
    for traj in predictions:
        if traj.completed_maneuver and not traj.crashed:
            successful += 1
    return successful / total

Common pitfalls

• Confusing training success rate with test success rate, as the paper notes dense rewards converge faster but sparse rewards may generalize differently.
• Assuming LiDAR & camera modalities are sufficient for roundabouts, as the paper shows they lack road curvature data and perform worse on openDD.
• Overlooking that success is defined by maneuver completion without crashes, not just proximity to a target.

Evidence (verbatim from paper)

Table III: Success rates on test scenarios for CRTS scenarios with bird’s-eye view input. Again we can compare the success rate from training (Figure 4) and test (Table III). The dense and no failure penalty look very similar during training, sparse reward takes more time to converge, but seems to reach the same performance. In the evaluation however, dense reward is the clear winner.

Citation

@misc{osinski2020carla,
  title={CARLA Real Traffic Scenarios -- novel training ground and benchmark for autonomous driving},
  author={Osiński et al. (2020)},
  year={2020},
  note={arXiv:2012.11329}
}

• arXiv: 2012.11329