Nurisk Eval

nurisk-eval

NuRisk: A Visual Question Answering Dataset for Agent-Level Risk Assessment in Autonomous Driving — Gao et al. (2025) (arXiv:2509.25944, 2025)

What this evaluates

Evaluates Vision-Language Models' ability to perform quantitative, agent-level risk assessment in autonomous driving. It probes spatio-temporal reasoning by testing whether models can predict collision risks, spatial distances, and temporal metrics based on visual sequences and optional physics-enhanced textual inputs.

Datasets

• NuRisk — total 1100000; splits: test (-1)

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error quantifying the average magnitude of error between predicted and ground-truth ordinal risk scores.
• QWK — range: [0, 1]
  • Quadratic Weighted Kappa evaluating the agreement between predicted and ground-truth risk levels, accounting for ordinal distance.
• Accuracy — range: [0, 1]
  • Standard classification accuracy representing the percentage of correct ordinal risk predictions.
• Spatial Accuracy — range: [0, 1]
  • Percentage of longitudinal and lateral distance predictions that fall within a 0.5-meter tolerance of the ground truth.
• Temporal Accuracy — range: [0, 1]
  • Percentage of Time-to-Collision (TTC) predictions that fall within a 0.5-second tolerance of the ground truth.
• Response Time — range: other
  • Average inference latency measured in milliseconds per instance.

Input / output format

Input: Image sequences (single or multi-frame) from autonomous driving scenarios, optionally augmented with textual physics-based information (position, velocity, acceleration of ego and nearby vehicles). Prompts vary by strategy (zero-shot, contextual prompting, chain-of-thought, in-context learning).

Output: Ordinal risk score prediction, spatial distance estimates (longitudinal/lateral), temporal estimates (TTC), and optionally reasoning steps if CoT is used.

Scoring recipe

def compute_metrics(predictions, ground_truth, pred_spatial, gt_spatial, pred_ttc, gt_ttc):
    mae = np.mean(np.abs(np.array(predictions) - np.array(ground_truth)))
    qwk = quadratic_weighted_kappa(ground_truth, predictions)
    acc = np.mean(np.array(predictions) == np.array(ground_truth))
    spatial_acc = np.mean(np.abs(np.array(pred_spatial) - np.array(gt_spatial)) <= 0.5)
    temporal_acc = np.mean(np.abs(np.array(pred_ttc) - np.array(gt_ttc)) <= 0.5)
    return {'MAE': mae, 'QWK': qwk, 'Accuracy': acc, 'Spatial Accuracy': spatial_acc, 'Temporal Accuracy': temporal_acc}

Common pitfalls

• Proprietary models achieve high accuracy only when physics text is provided, indicating reliance on textual priors rather than visual-spatio-temporal reasoning.
• Advanced prompting strategies (CoT, ICL) increase latency without improving accuracy for pre-trained models, as external templates do not enhance internal reasoning for this domain.
• Open-source models suffer from context window limitations when processing token-heavy physics inputs, causing drastic latency spikes.

Evidence (verbatim from paper)

VLM performance is assessed across three categories. For risk assessment accuracy, we use Mean Absolute Error (MAE) to quantify the average magnitude of error in the ordinal risk predictions, Quadratic Weighted Kappa (QWK) to evaluate the agreement between predicted and ground-truth risk levels, standard Accuracy (Acc), and Precision, Recall, F1-Score for agent risk score analysis. For spatio-temporal reasoning, we measure Spatial Accuracy (percentage of longitudinal and lateral predictions within a 0.5-meter tolerance) and Temporal Accuracy (TTC) (percentage of longitudinal and lateral predictions within a 0.5-second tolerance).

Citation

@misc{gao2025nurisk,
  title={NuRisk: A Visual Question Answering Dataset for Agent-Level Risk Assessment in Autonomous Driving},
  author={Gao et al. (2025)},
  year={2025},
  note={arXiv:2509.25944}
}

• arXiv: 2509.25944