Carla Urban Driving Eval

carla-urban-driving-eval

Dynamic Conditional Imitation Learning for Autonomous Driving — Eraqi et al. (2022) (arXiv:2211.11579, 2022)

What this evaluates

Evaluates an autonomous driving agent's ability to navigate urban environments, avoid dynamic and static obstacles, and handle road blockages under varying weather conditions and unseen towns.

Datasets

• CARLA urban driving benchmark — total 1200; splits: test (1200)

Metrics

• Success Rate (primary) — range: percent
  • Percentage of test scenarios where the vehicle reaches the destination within a predetermined deadline (set to the time needed to traverse the shortest route at 10 km/h).
• Distance to Goal Traveled — range: percent
  • Average percentage of the total distance to the destination covered by the vehicle across all test scenarios.
• Average Kilometers Before Infraction — range: km
  • Mean distance (in km) driven until the first occurrence of a specific infraction (e.g., collision with pedestrian/vehicle/static object, going off-road, invading opposite lane, violating traffic light).

Input / output format

Input: RGB camera images, 32-layer LiDAR point clouds, GPS/waypoint coordinates, traffic light states, and vehicle kinematics from the CARLA simulator.

Output: Continuous driving control commands (steering, throttle, brake) and high-level navigational commands (e.g., 'follow lane', 'go left').

Scoring recipe

def score_eval(scenarios, predictions, gold):
    successes = 0
    dist_to_goal_sum = 0.0
    infraction_dists = {k: [] for k in gold['infraction_types']}
    for i, scenario in enumerate(scenarios):
        traj = predictions[i]
        if reached_destination(traj) and within_deadline(traj, gold['deadline']):
            successes += 1
        dist_to_goal_sum += calc_dist_to_goal_pct(traj, gold['dest'])
        for inf_type in infraction_dists:
            dist = get_first_infraction_dist(traj, inf_type)
            infraction_dists[inf_type].append(dist if dist > 0 else 0.0)
    success_rate = (successes / len(scenarios)) * 100
    avg_dist_to_goal = dist_to_goal_sum / len(scenarios)
    avg_infraction_km = {k: np.mean(v) for k, v in infraction_dists.items()}
    return success_rate, avg_dist_to_goal, avg_infraction_km

Common pitfalls

• CARLA simulator non-determinism (texture loading, pedestrian algorithms) causes result variance across runs.
• The success deadline is set to a low speed (10 km/h) shortest route time, allowing faster models to succeed even if they deviate from the optimal path.
• Results may not exactly match prior publications due to CARLA version updates and rendering changes.

Evidence (verbatim from paper)

A test scenario is considered successful if the vehicle reaches the destination within a predetermined deadline. The deadline (maximum allowed time to reach the destination) is set to the time needed to reach the destination along the shortest route at a low speed of 10 km/h as followed in [[14]] and [[10]]. ... The table reports the autonomous driving success rate on different tasks and test conditions, and the average percentage of distance to goal traveled is available between parentheses.

Citation

@misc{eraki2022dynamiccil,
  title={Dynamic Conditional Imitation Learning for Autonomous Driving},
  author={Eraqi et al. (2022)},
  year={2022},
  note={arXiv:2211.11579}
}

• arXiv: 2211.11579