impromptu-vla-diagnostic-eval
Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models — Chi et al. (2025) (arXiv:2505.23757, 2025)
What this evaluates
Diagnoses VLM capabilities in autonomous driving by evaluating perception, prediction, meta-planning via Q&A accuracy, and planning via trajectory prediction L2 error.
Datasets
Metrics
Q&A Accuracy (primary) — range: [0, 1]
- Percentage of correctly answered questions across four categories: V.R.U. (Vulnerable Road Users), T. Light (Traffic Lights), Dyn. Obj. (Dynamic Objects), and M.P. (Meta-Planning).
Traj. Pred. L2 Error (m) — range: m
- Euclidean distance in meters between predicted and ground truth trajectories at 1s, 2s, 3s, and 4s horizons, plus average across horizons.
Input / output format
Input: Video clips with planning-oriented Q&A prompts and trajectory ground truth.
Output: Text answers to Q&A and predicted trajectory points.
Scoring recipe
def score_diagnostic(predictions, gold):
qa_acc = sum(1 for p, g in zip(predictions['qa'], gold['qa']) if p == g) / len(gold['qa'])
traj_errors = []
for horizon in [1, 2, 3, 4]:
pred_pts = predictions['traj'][horizon]
gt_pts = gold['traj'][horizon]
l2 = np.sqrt(np.sum((pred_pts - gt_pts)**2, axis=1))
traj_errors.append(np.mean(l2))
return qa_acc, traj_errors, np.mean(traj_errors)
Common pitfalls
- Q&A accuracy is broken down into four distinct capability categories rather than a single aggregate score.
- Trajectory prediction horizon extends to 4s in this diagnostic set, differing from the 3s horizon used in the nuScenes open-loop benchmark.
Evidence (verbatim from paper)
The quantitative evaluation on the Impromptu VLA validation set, summarized in Table [3], clearly demonstrates that fine-tuning on our dataset can transforms to all crucial aspects of autonomous driving, including perception, prediction, reasoning for planning, and the planned trajectory. Accuracy ↑ is reported for perception (V.R.U., T. Light), prediction (Dyn. Obj.), meta-planning (M.P.) and Planning (L2).
Citation
@misc{chi2025impromptu,
title={Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models},
author={Chi et al. (2025)},
year={2025},
note={arXiv:2505.23757}
}
1---2name: impromptu-vla-diagnostic-eval3description: Diagnoses VLM capabilities in autonomous driving by evaluating perception, prediction, meta-planning via Q&A accuracy, and planning via trajectory prediction L2 error. Use when the user wants to benchmark on Impromptu VLA, or asks about evaluating this task. Reports Q&A Accuracy.4---56# impromptu-vla-diagnostic-eval78> Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models — Chi et al. (2025) (arXiv:2505.23757, 2025)910## What this evaluates1112Diagnoses VLM capabilities in autonomous driving by evaluating perception, prediction, meta-planning via Q&A accuracy, and planning via trajectory prediction L2 error.1314## Datasets1516- **Impromptu VLA** — total 80000; splits: validation (-1); repo https://github.com/ahydchh/Impromptu-VLA1718## Metrics1920- `Q&A Accuracy` **(primary)** — range: [0, 1]21 - Percentage of correctly answered questions across four categories: V.R.U. (Vulnerable Road Users), T. Light (Traffic Lights), Dyn. Obj. (Dynamic Objects), and M.P. (Meta-Planning).22- `Traj. Pred. L2 Error (m)` — range: m23 - Euclidean distance in meters between predicted and ground truth trajectories at 1s, 2s, 3s, and 4s horizons, plus average across horizons.2425## Input / output format2627**Input**: Video clips with planning-oriented Q&A prompts and trajectory ground truth.2829**Output**: Text answers to Q&A and predicted trajectory points.3031## Scoring recipe3233```python34def score_diagnostic(predictions, gold):35 qa_acc = sum(1 for p, g in zip(predictions['qa'], gold['qa']) if p == g) / len(gold['qa'])36 traj_errors = []37 for horizon in [1, 2, 3, 4]:38 pred_pts = predictions['traj'][horizon]39 gt_pts = gold['traj'][horizon]40 l2 = np.sqrt(np.sum((pred_pts - gt_pts)**2, axis=1))41 traj_errors.append(np.mean(l2))42 return qa_acc, traj_errors, np.mean(traj_errors)43```4445## Common pitfalls4647- Q&A accuracy is broken down into four distinct capability categories rather than a single aggregate score.48- Trajectory prediction horizon extends to 4s in this diagnostic set, differing from the 3s horizon used in the nuScenes open-loop benchmark.4950## Evidence (verbatim from paper)5152> The quantitative evaluation on the Impromptu VLA validation set, summarized in Table [3], clearly demonstrates that fine-tuning on our dataset can transforms to all crucial aspects of autonomous driving, including perception, prediction, reasoning for planning, and the planned trajectory. Accuracy ↑ is reported for perception (V.R.U., T. Light), prediction (Dyn. Obj.), meta-planning (M.P.) and Planning (L2).5354## Citation5556```bibtex57@misc{chi2025impromptu,58 title={Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models},59 author={Chi et al. (2025)},60 year={2025},61 note={arXiv:2505.23757}62}63```6465- arXiv: 2505.23757
qhjqhj00