Robodrivebench Eval

robodrivebench-eval

RoboDriveVLM: A Novel Benchmark and Baseline towards Robust Vision-Language Models for Autonomous Driving — Dacheng Liao et al. (arXiv:2512.01300, 2025)

What this evaluates

Evaluates the robustness and safety of vision-language models (VLMs) for end-to-end autonomous driving when subjected to real-world sensor corruptions (e.g., fog, rain, motion blur) and prompt corruptions (e.g., bit errors, malicious attacks). It probes the model's ability to maintain accurate trajectory prediction and low collision rates under degraded inputs.

Datasets

• RoboDriveBench — total 64559; splits: test (-1)

Metrics

• AvgL2 (primary) — range: meters
  • Average L2 loss (trajectory prediction error) computed over three severity levels of corruption.
• AvgCol — range: percent
  • Average collision rate computed over three severity levels of corruption.
• MCL2 — range: percent
  • Mean Corruption L2, reported as a percentage, aggregating L2 errors across all corruption scenarios relative to the clean baseline.
• MCC — range: percent
  • Mean Corruption Collision, reported as a percentage, aggregating collision rates across all corruption scenarios relative to the clean baseline.
• Inv. — range: count
  • Invalid Prediction Count, tallying outputs that fail to produce a valid trajectory or control command.

Input / output format

Input: Multimodal inputs including camera images, LiDAR point clouds, and radar data, combined with language prompts. Historical trajectory data and consecutive image frames or vehicle speed/curvature sequences are also provided depending on the model architecture.

Output: Predicted future driving trajectory (and optionally future speed and curvature values).

Scoring recipe

def compute_metrics(predictions, gold, corruption_types):
    l2_errors = [l2_dist(p, g) for p, g in zip(predictions, gold)]
    collisions = [1 if e > COLLISION_THRESHOLD else 0 for e in l2_errors]
    avg_l2 = mean(l2_errors)
    avg_col = mean(collisions) * 100
    clean_l2 = mean([e for e, c in zip(l2_errors, corruption_types) if c == 'clean'])
    clean_col = mean([c for c, corr in zip(collisions, corruption_types) if corr == 'clean'])
    mcl2 = mean([e for e, c in zip(l2_errors, corruption_types) if c != 'clean']) / clean_l2 * 100
    mcc = mean([c for c, corr in zip(collisions, corruption_types) if corr != 'clean']) / clean_col * 100
    inv = sum(1 for p in predictions if is_invalid(p))
    return avg_l2, avg_col, mcl2, mcc, inv

Common pitfalls

• MCL2 and MCC are reported as percentages that can exceed 100%, indicating a relative degradation in performance compared to the clean baseline rather than an absolute rate.
• Invalid predictions (Inv.) are counted separately and may be excluded from AvgL2 and collision rate calculations, meaning reported error metrics might not reflect the full impact of severe prompt corruption.

Evidence (verbatim from paper)

The main experimental results are presented across four tables. Figure 3 summarizes the average performance of all models across four key evaluation metrics: AvgL2, AvgCol, MCL2, and MCC, under both sensor corruption and prompt corruption.

Citation

@misc{liao2025robodrivevlm,
  title={RoboDriveVLM: A Novel Benchmark and Baseline towards Robust Vision-Language Models for Autonomous Driving},
  author={Dacheng Liao et al.},
  year={2025},
  note={arXiv:2512.01300}
}

• arXiv: 2512.01300