Racecar Eval

racecar-eval

RACECAR -- The Dataset for High-Speed Autonomous Racing — Kulkarni et al. (2023) (arXiv:2306.03252, 2023)

What this evaluates

Evaluates high-speed autonomous driving capabilities including precise localization, long-range object detection and tracking, and robust mapping/SLAM under extreme dynamic conditions (up to 170 mph). The protocol benchmarks how well models maintain accuracy and latency when processing multi-modal sensor data at racing speeds where motion blur, sensor dropout, and rapid ego-motion are prevalent.

Datasets

• RACECAR — total ?; splits: train (-1), test (-1); repo https://github.com/linklab-uva/RACECAR_DATA

Metrics

• Average Precision (AP) (primary) — range: percent
  • Standard object detection metric computing the area under the precision-recall curve. The paper reports AP averaged over classes, evaluated separately across four distance ranges (0-20m, 20-40m, 40-60m, 60m-Inf) to capture high-speed range degradation.
• Localization Error — range: meters
  • Measured as the distance the vehicle travels between GNSS updates or the deviation of the fused pose estimate from ground truth. Reported in meters to quantify blind-spot distance and tracking stability at 100+ mph.

Input / output format

Input: Synchronized multi-modal sensor streams including 3D LiDAR point clouds, radar detections (angle, distance, velocity), camera images (6 global shutter cameras), GNSS coordinates, and IMU data (accelerometer/gyroscope at 125 Hz, wheel rotation at 100 Hz).

Output: For detection: 3D bounding boxes or BEV detections with class labels, confidence scores, and distance estimates. For localization: vehicle pose [x, y, z, θ] in world frame at 100 Hz. For mapping: 3D point cloud map or pose trajectory with motion distortion compensation.

Scoring recipe

def compute_ap(predictions, ground_truth, iou_thresh=0.5):
    aps = {}
    ranges = [(0, 20), (20, 40), (40, 60), (60, float('inf'))]
    for low, high in ranges:
        pred_f = [p for p in predictions if low <= p['distance'] < high]
        gt_f = [g for g in ground_truth if low <= g['distance'] < high]
        precisions, recalls = compute_pr_curve(pred_f, gt_f, iou_thresh)
        ap = interpolate_ap(precisions, recalls)
        aps[f'{low}-{high}m'] = ap
    return {'overall_ap': sum(aps.values())/len(aps), 'range_aps': aps}

Common pitfalls

• High-speed motion blur and LiDAR dropout at 100+ mph significantly degrade detection and mapping performance; models must compensate for scan distortion before matching.
• GNSS-only localization updates at 20 Hz are insufficient for racing; EKF fusion at 100 Hz is required to reduce blind-spot distance from ~2m to ~0.4m.
• Distance-dependent performance drops sharply beyond 60m; reporting overall AP without range breakdown hides critical high-speed safety limitations.

Evidence (verbatim from paper)

Inference on point clouds using PointPillars resulted in maximum detections at distances up to 110 m. However, the average precision of detections dropped approximately 40% for distances over 60 m. A benchmark challenge is to improve the detection range and reliability.

Citation

@misc{kulkarni2023racecar,
  title={RACECAR -- The Dataset for High-Speed Autonomous Racing},
  author={Kulkarni et al. (2023)},
  year={2023},
  note={arXiv:2306.03252}
}

• arXiv: 2306.03252