Neuroncap Closed Loop Eval

neuroncap-closed-loop-eval

Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models — Chi et al. (2025) (arXiv:2505.23757, 2025)

What this evaluates

Evaluates closed-loop autonomous driving performance in simulated real-world scenarios, measuring safety and planning efficiency through collision avoidance and impact speed mitigation.

Datasets

• nuScenes — total ?; splits: test (-1)

Metrics

• NeuroNCAP Score (NNS) (primary) — range: [0, 5]
  • 5.0 if no collision occurs; otherwise 4.0 * max(0, 1 - v_i / v_r), where v_i is actual impact speed and v_r is reference impact speed without evasive action. Capped at 4.0 for collisions.
• Collision rate (%) — range: percent
  • Percentage of scenarios resulting in a collision, categorized by interaction type (Avg, Stat, Frontal, Side).

Input / output format

Input: Video frames and sensor data from nuScenes driving scenarios.

Output: Vehicle control actions (steering, acceleration, braking) over time.

Scoring recipe

def score_neuroncap(predictions, gold):
    collisions = [p['collides'] for p in predictions]
    collision_rate = sum(collisions) / len(collisions) * 100
    scores = []
    for p in predictions:
        if not p['collides']:
            scores.append(5.0)
        else:
            vi = p['impact_speed']
            vr = p['reference_speed']
            scores.append(4.0 * max(0, 1 - vi / vr))
    return sum(scores) / len(scores), collision_rate

Common pitfalls

• NNS maxes at 5.0 only for zero collisions; collision scenarios are capped at 4.0.
• Metrics are reported across four interaction categories (Avg, Stat, Frontal, Side), not just a single global average.

Evidence (verbatim from paper)

The NNS is computed, in the spirit of a 5-star rating system, as follows: a score of 5.0 is achieved if no collision occurs; otherwise, the score is 4.0⋅max(0,1−vi/vr), where vi is the actual impact speed (magnitude of relative velocity between the ego-vehicle and the colliding actor) and vr is the reference impact speed that would occur if no evasive action were performed. Collision rates, on the other hand, directly track the percentage of scenarios resulting in a collision.

Citation

@misc{chi2025impromptu,
  title={Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models},
  author={Chi et al. (2025)},
  year={2025},
  note={arXiv:2505.23757}
}

• arXiv: 2505.23757