Ads Violation Cause Eval

ads-violation-cause-eval

Towards Automated Driving Violation Cause Analysis in Scenario-Based Testing for Autonomous Driving Systems — Wan et al. (2024) (arXiv:2401.10443, 2024)

What this evaluates

Evaluates an automated root-cause analysis tool for autonomous driving systems by measuring its ability to correctly identify the faulty component and the specific output message that caused a driving violation in simulation. It also measures the debugging scope reduction and computational efficiency of the tool.

Datasets

• ADS Violation Cause Benchmark — total ?; splits: test (-1)

Metrics

• component-level success (primary) — range: [0, 1]
  • Boolean accuracy indicating whether the tool's reported component matches the ground-truth faulty component.
• message-level success — range: [0, 1]
  • Boolean accuracy indicating whether the tool's reported output message matches the ground-truth faulty message.
• reduction_rate — range: percent
  • Defined as 1 - 1/|M|, where |M| is the total number of messages in the violation-included driving trace. Drops to 0% if message-level success fails.
• attribution_time — range: other
  • Wall-clock time in minutes required for the tool to complete the cause attribution task.

Input / output format

Input: Simulation traces and inter-component messages from Baidu Apollo 7.0 running in the LGSVL simulator under predefined collision scenarios.

Output: A tuple containing (1) the identified faulty component name, and (2) a specific input/output message pair from that component.

Scoring recipe

def compute_metrics(predictions, gold):
    comp_success = 1.0 if predictions.component == gold.component else 0.0
    msg_success = 1.0 if predictions.msg_pair == gold.msg else 0.0
    total_msgs = len(gold.trace_messages)
    reduction = (1.0 - 1.0 / total_msgs) if msg_success else 0.0
    return comp_success, msg_success, reduction

Common pitfalls

• Simulation replay randomness can cause vehicle state drift, making exact message matching difficult and leading to near-miss timestamp matches instead of exact failures.
• The reduction rate metric drops to 0% for any instance where message-level success fails, heavily penalizing partial matches.
• Real bug scenarios require manual reconstruction from GitHub PRs/commits, introducing potential variability in ground-truth alignment.

Evidence (verbatim from paper)

We use two boolean metrics to report the effectiveness of the cause localization task: (1) component-level success: which is defined as whether the reported component is the component that contains a real bug/fault, and (2) message-level success: whether the reported output message is the result of the real bug/injected fault.

Citation

@misc{wan2024towards,
  title={Towards Automated Driving Violation Cause Analysis in Scenario-Based Testing for Autonomous Driving Systems},
  author={Wan et al. (2024)},
  year={2024},
  note={arXiv:2401.10443}
}

• arXiv: 2401.10443