Drivecritic Eval

drivecritic-eval

DriveCritic: Towards Context-Aware, Human-Aligned Evaluation for Autonomous Driving with Vision-Language Models — Jingyu Song et al. (2025) (arXiv:2510.13108, 2025)

What this evaluates

Evaluates a model's ability to judge autonomous driving trajectory pairs based on context-aware reasoning, safety, and human preferences, rather than relying on rigid rule-based thresholds. It probes whether the model can integrate visual and symbolic context to reason about nuanced traffic situations like lateral buffer maintenance or stop sign compliance.

Datasets

• DriveCritic — total ?; splits: train (1100), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of pairwise comparisons where the model's predicted preferred trajectory matches the human-preferred trajectory.
• Robustness Rate (RR) — range: [0, 1]
  • Fraction of test pairs where the model's prediction remains unchanged after swapping the order of the two trajectories in the prompt: RR = (1/|D|) * sum(I[y^i == y_hat^i]).

Input / output format

Input: A prompt containing visual context (stitched camera images and BEV maps), ego vehicle status, and two candidate trajectory waypoints (Trajectory A and Trajectory B) for a 4-second horizon.

Output: A single judgment indicating which of the two trajectories (A or B) is preferred by the model.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
    return correct / len(predictions)

def compute_robustness_rate(original_preds, flipped_preds):
    consistent = sum(1 for p, fp in zip(original_preds, flipped_preds) if p == fp)
    return consistent / len(original_preds)

Common pitfalls

• Models often exhibit position bias, preferring the first trajectory listed in the prompt regardless of actual quality.
• Rule-based metrics like EPDMS rely on fixed thresholds and fail to capture nuanced social norms or safety trade-offs in ambiguous scenarios.

Evidence (verbatim from paper)

The primary evaluation metric is accuracy, defined as the proportion of pairwise comparisons in which the model’s judgment agrees with the human-preferred trajectory. ... To quantify robustness, we perform a position-flip test: for every test pair, we swap the order of Trajectory A and Trajectory B in the prompt and re-evaluate the model. We follow[[36]] to compute the Robustness Rate (RR): RR = (1/|D|)∑ I[y^i = ŷ^i]

Citation

@misc{song2025drivecritic,
  title={DriveCritic: Towards Context-Aware, Human-Aligned Evaluation for Autonomous Driving with Vision-Language Models},
  author={Jingyu Song et al. (2025)},
  year={2025},
  note={arXiv:2510.13108}
}

• arXiv: 2510.13108