Nuscenes Openloop Trajectory Eval

nuscenes-openloop-trajectory-eval

Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models — Chi et al. (2025) (arXiv:2505.23757, 2025)

What this evaluates

Assesses open-loop trajectory prediction accuracy of VLMs by measuring the distance between predicted and ground truth future paths at multiple time horizons.

Datasets

• nuScenes — total ?; splits: test (-1)

Metrics

• L2 Error (m) (primary) — range: m
  • Euclidean distance in meters between predicted and ground truth trajectory points at 1s, 2s, and 3s horizons. Average L2 Error is computed across these horizons.

Input / output format

Input: Historical trajectory and scene context from nuScenes.

Output: Predicted future trajectory points (x, y coordinates) for 1s, 2s, and 3s horizons.

Scoring recipe

def score_l2(predictions, gold):
    errors = []
    for horizon in [1, 2, 3]:
        pred_pts = predictions[horizon]
        gt_pts = gold[horizon]
        l2 = np.sqrt(np.sum((pred_pts - gt_pts)**2, axis=1))
        errors.append(np.mean(l2))
    return errors, np.mean(errors)

Common pitfalls

• Evaluation is strictly open-loop; models do not receive feedback from the environment during prediction.
• Horizons are fixed at 1s, 2s, and 3s; average L2 is the mean across these three specific horizons.

Evidence (verbatim from paper)

Performance is primarily measured by the L2 distance (in meters) between the predicted and ground truth trajectories at future time horizons of 1s, 2s, and 3s, along with the average L2 error.

Citation

@misc{chi2025impromptu,
  title={Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models},
  author={Chi et al. (2025)},
  year={2025},
  note={arXiv:2505.23757}
}

• arXiv: 2505.23757