Carla Dos Eval

carla-dos-eval

ReasonNet: End-to-End Driving with Temporal and Global Reasoning — Hao Shao et al. (2023) (arXiv:2305.10507, 2023)

What this evaluates

Evaluates end-to-end autonomous driving agents on route completion, safety/infractions, and occlusion-aware perception in simulated urban environments.

Datasets

• CARLA Town 05 Long — total ?; splits: test (-1)
• DOS Benchmark — total ?; splits: test (-1)

Metrics

• Driving Score (DS) (primary) — range: [0, 1]
  • DS = RC × IS. Describes both driving progress and safety, and is the primary ranking metric.
• Route Completion (RC) — range: [0, 1]
  • RC = completed_route_distance / total_route_distance. Percentage of the route completed.
• Infraction Score (IS) — range: [0, 1]
  • IS = 1.0 × discount_factor^(number_of_collisions_or_violations). Decays when collisions or traffic rule violations occur.
• Collision Rate (CR) — range: other
  • CR = total_collisions / driven_distance_km. Normalized by driven distance.
• Red Light Violation (Red) — range: other
  • Red = red_light_violations / driven_distance_km. Normalized by driven distance.
• Blocked Frequency (Blocked) — range: other
  • Blocked = blocked_events / driven_distance_km. Normalized by driven distance.
• Success Rate (SR) — range: [0, 1]
  • SR = successful_scenario_runs / total_scenario_runs. Evaluated per scenario type (SR#1 to SR#4) on the DOS benchmark.

Input / output format

Input: Multi-view camera images and 3D point cloud data from CARLA simulator, processed via ResNet-50 (2D) and PointPillars (3D) backbones.

Output: Waypoints and occupancy map predictions for $T_f=4$ future time steps at 0.5s intervals.

Scoring recipe

def compute_metrics(route, driven, events):
    rc = driven.distance / route.total_distance
    is_score = 1.0
    for e in events:
        if e.type in ['collision', 'traffic_violation']:
            is_score *= discount_factor
    ds = rc * is_score
    cr = events.collision_count / driven.distance_km
    red = events.red_violations / driven.distance_km
    blocked = events.blocked_count / driven.distance_km
    sr = successful_runs / total_runs_per_scenario
    return ds, rc, is_score, cr, red, blocked, sr

Common pitfalls

• DS is the product of RC and IS, so optimizing for route completion alone can mask poor safety performance.
• Success rates on the DOS benchmark are highly sensitive to long-term memory; removing it drops performance by ~46% on occlusion scenarios.
• Metrics are reported as mean ± std over 3 independent runs, not single-pass results.

Evidence (verbatim from paper)

We consider three major metrics introduced by the CARLA LeaderBoard: route completion ratio (RC), infraction score (IS), and driving score (DS). The route completion ratio is the percentage of the route completed. The infraction score measures infractions triggered. When collisions or traffic rule violations occur, the infraction score will decay by a discount factor. The driving score is the product of the route completion ratio and the infraction score, describing both driving progress and safety, and thus is the primary ranking metric in the CARLA Leaderboard.

Citation

@misc{shao2023reasonnet,
  title={ReasonNet: End-to-End Driving with Temporal and Global Reasoning},
  author={Hao Shao et al. (2023)},
  year={2023},
  note={arXiv:2305.10507}
}

• arXiv: 2305.10507