Carla Longest6 Town05long Eval

carla-longest6-town05long-eval

Sensor Fusion by Spatial Encoding for Autonomous Driving — Lai-Dang et al. (2023) (arXiv:2308.10707, 2023)

What this evaluates

Evaluates an autonomous driving model's ability to navigate complex urban and highway environments under varying weather and lighting conditions, focusing on route completion, safety (infraction avoidance), and overall driving performance.

Datasets

• CARLA Longest6 — total ?; splits: test (-1)
• CARLA Town05 Long — total ?; splits: test (-1)

Metrics

• Driving Score (DS) (primary) — range: [0, 1] | percent
  • DS = RC × IS, where RC is the percentage of the route completed and IS is a safety score that decreases with infractions. Additional per-km infraction counts are reported for pedestrians, vehicles, static objects, red lights, off-road, route deviation, timeout, and blocking.
• Route Completion (RC) — range: percent
  • Percentage of the total route distance successfully traversed without terminating the episode.
• Infraction Score (IS) — range: [0, 1]
  • Safety metric that decreases when traffic rules or collision constraints are violated; typically normalized to [0, 1].

Input / output format

Input: Synchronized camera images and LiDAR point clouds collected from junctions and curved highways in the CARLA simulator, along with ego-vehicle state.

Output: Continuous vehicle control commands (steering, throttle, brake) and discrete gear selections to navigate the predefined route.

Scoring recipe

def compute_metrics(predictions, gold_route):
    rc = len(predictions.completed_path) / len(gold_route.total_path)
    infractions = count_violations(predictions, gold_route)
    is_score = max(0.0, 1.0 - (infractions / gold_route.max_infractions))
    ds = rc * is_score
    return {"RC": rc, "IS": is_score, "DS": ds}

Common pitfalls

• Confusing Driving Score (DS) with Route Completion (RC); DS heavily penalizes safety violations, so a high RC does not guarantee a high DS.
• Failing to account for the combined weather and daylight conditions in Longest6, which drastically changes sensor fusion performance and requires robust multi-resolution encoding.
• Reporting only aggregate scores without per-kilometer infraction breakdowns, which obscures specific safety failures like pedestrian or static collisions.

Evidence (verbatim from paper)

The proposed method is evaluated using three metrics: route completion (RC), infraction score (IS), and driving score (DS). RC measures the percentage of the route completed, IS decreases when infractions occur, and DS is a comprehensive metric that considers both progress and safety.

Citation

@misc{lai2023sensorfusion,
  title={Sensor Fusion by Spatial Encoding for Autonomous Driving},
  author={Lai-Dang et al. (2023)},
  year={2023},
  note={arXiv:2308.10707}
}

• arXiv: 2308.10707