V2v QA Eval

v2v-qa-eval

V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multi-Modal Large Language Models — Chiu et al. (2025) (arXiv:2502.09980, 2025)

What this evaluates

Evaluates a multi-modal LLM's ability to fuse 3D perception features from multiple connected vehicles to answer safety-critical driving queries. It probes spatial grounding, notable object identification near planned waypoints, and collision-avoidance trajectory planning in cooperative autonomous driving scenarios.

Datasets

• V2V-QA — total ?; splits: V2V-split (-1), V2X-split (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for spatial grounding and object identification tasks. Computed per question type (Q1-Q4) and averaged for QGr.
• L2 distance error — range: other
  • Euclidean distance between predicted future trajectory waypoints and ground-truth trajectory waypoints, measured in meters.
• Collision rate — range: percent
  • Percentage of generated planning trajectories that result in a collision with nearby objects or obstacles.
• Communication cost — range: other
  • Total data transferred per timestep between CAVs and the centralized LLM node, calculated as (0.203 + 0.0004 * N_q) MB per CAV, where N_q is the number of questions.

Input / output format

Input: Scene-level feature maps and object-level feature vectors extracted from LiDAR point clouds of multiple connected autonomous vehicles (CAVs), combined with a natural language question specifying grounding references, object identification targets, or planning waypoints.

Output: Natural language response containing predicted spatial coordinates for grounding, identified notable objects, and future driving trajectory waypoints.

Scoring recipe

def compute_metrics(predictions, gold):
    # Grounding & Identification (Q1-Q4)
    f1_scores = []
    for pred_locs, gold_locs in zip(predictions, gold):
        tp = count_spatial_matches(pred_locs, gold_locs, threshold=0.5)
        prec = tp / (len(pred_locs) + 1e-8)
        rec = tp / (len(gold_locs) + 1e-8)
        f1_scores.append(2 * prec * rec / (prec + rec + 1e-8))
    f1 = mean(f1_scores)
    
    # Planning (Q5)
    l2_errors = [np.linalg.norm(p - g) for p, g in zip(predictions, gold)]
    l2 = mean(l2_errors)
    cr = count_collisions(predictions) / len(predictions)
    
    return {'F1': f1, 'L2': l2, 'CR': cr}

Common pitfalls

• F1 scores for grounding tasks rely on implicit spatial matching thresholds not explicitly defined in the main text, making exact replication difficult without supplementary code.
• Planning metrics (L2 distance and collision rate) require specific trajectory alignment and collision detection logic that varies across implementations.
• Communication cost scales non-linearly with the number of vehicles (N_v) and questions (N_q), so direct comparisons are sensitive to these hyperparameters rather than reflecting pure model efficiency.

Evidence (verbatim from paper)

Q1: Grounding at a reference location. Q2: Grounding behind a reference object at a location. Q3: Grounding behind a reference object in a direction. QGr: Average of grounding (Q1, Q2, and Q3). Q4: Notable object identification. Q5: Planning. L2: L2 distance error. CR: Collision rate.

Citation

@misc{chiu2025v2vllm,
  title={V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multi-Modal Large Language Models},
  author={Chiu et al. (2025)},
  year={2025},
  note={arXiv:2502.09980}
}

• arXiv: 2502.09980