Carla Autonomous Driving Eval

carla-autonomous-driving-eval

HPRM: High-Performance Robotic Middleware for Intelligent Autonomous Systems — Kwok et al. (2024) (arXiv:2412.01799, 2024)

What this evaluates

Evaluates the end-to-end inference latency of an autonomous driving pipeline that runs parallel reinforcement learning and object detection models in a simulated urban environment. It measures how efficiently the middleware handles communication and computation overhead during real-time sensor processing and action fusion.

Datasets

• CARLA simulator — total ?; splits: test (400)

Metrics

• inference_latency (primary) — range: ms
  • Sum of communication time and inference time per frame, explicitly excluding physics computation time.

Input / output format

Input: BEV and RGB camera images from the CARLA simulator, fed into parallel PPO policy and YOLO object detection models.

Output: Final control action determined by a Fusion reactor/node after synchronizing and combining outputs from the PPO and YOLO models.

Scoring recipe

latencies = []
for step in range(100):  # warm-up
    run_pipeline()
for step in range(400):  # benchmark
    t_start = time.now()
    action = run_pipeline()
    t_end = time.now()
    latencies.append(t_end - t_start - physics_time)
return mean(latencies)

Common pitfalls

• Including CARLA physics computation time in the latency measurement, which inflates results and misrepresents middleware performance.
• Failing to discard the initial 100 warm-up steps, leading to skewed latency due to cold-start effects and JIT/model loading overhead.
• Comparing centralized and decentralized coordination without accounting for synchronization overhead differences that diminish as object size grows.

Evidence (verbatim from paper)

Inference latency is measured as the sum of communication time and inference time. We found that running PPO policy inference in CPU and YOLO in GPU led to a slight performance increase due to full utilization of compute resources, and was implemented across the benchmark. The box plot in Figure [6] illustrates the inference latency measured when running the CARLA benchmark across 400 environment step frames after 100 warm-up steps with HPRM and ROS2 Humble. To obtain a more accurate measure of inference latency, our benchmarks exclude the time CARLA spends computing physics.

Citation

@misc{kwok2024hprm,
  title={HPRM: High-Performance Robotic Middleware for Intelligent Autonomous Systems},
  author={Kwok et al. (2024)},
  year={2024},
  note={arXiv:2412.01799}
}

• arXiv: 2412.01799