Drivelmmo1 Eval

drivelmmo1-eval

DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding — Ishaq et al. (2025) (arXiv:2503.10621, 2025)

What this evaluates

Evaluates step-by-step visual reasoning capabilities of multimodal models in autonomous driving scenarios, covering perception, prediction, and planning. It assesses both the logical coherence of intermediate reasoning steps and the accuracy of final answers.

Datasets

• DriveLMM-o1 — total 22000; splits: train (18000), test (4000); repo https://github.com/ayesha-ishaq/DriveLMM-o1

Metrics

• final reasoning score (primary) — range: [0, 1]
  • Average of 12 sub-metrics: Risk Assessment Accuracy, Traffic Rule Adherence, Scene Awareness and Object Understanding, Relevance, Missing Details, Faithfulness-Step, Informativeness-Step, Repetition-Token, Hallucination, Semantic Coverage-Step, Commonsense, and Missing Step. Each sub-metric is scored by GPT-4o against ground truth reasoning steps using a structured rubric.
• accuracy of the final answer — range: [0, 1]
  • Exact match accuracy for multiple-choice questions (MCQs) comparing the model's final answer to the ground truth.

Input / output format

Input: Multiview images, LiDAR point clouds, and keyframes from NuScenes, paired with a standardized question from a 10-question template covering perception, prediction, and planning.

Output: A step-by-step reasoning chain detailing observations, entity relationships, and outcomes, followed by a final answer (typically multiple-choice).

Scoring recipe

def evaluate_drivelmmo1(predictions, ground_truth):
    sub_metrics = [
        'Risk Assessment Accuracy', 'Traffic Rule Adherence',
        'Scene Awareness and Object Understanding', 'Relevance',
        'Missing Details', 'Faithfulness-Step', 'Informativeness-Step',
        'Repetition-Token', 'Hallucination', 'Semantic Coverage-Step',
        'Commonsense', 'Missing Step'
    ]
    scores = []
    for pred, gt in zip(predictions, ground_truth):
        prompt = build_evaluation_prompt(pred, gt, sub_metrics)
        llm_response = call_gpt4o(prompt)
        metric_values = parse_json_output(llm_response)
        scores.append(sum(metric_values) / len(metric_values))
    final_reasoning_score = sum(scores) / len(scores)
    accuracy = exact_match(predictions.final_answer, ground_truth.final_answer)
    return {'final_reasoning_score': final_reasoning_score, 'accuracy': accuracy}

Common pitfalls

• Using reference-free evaluation methods (like ROSCOE or ReCeval) instead of the required ground-truth reference scoring.
• Failing to align the model's step-by-step reasoning with the specific logical progression expected in autonomous driving safety contexts.
• Ignoring the multimodal inputs (LiDAR + multiview images) and treating the task as text-only VQA.

Evidence (verbatim from paper)

We borrow the Faithfulness-Step, Informativeness-Step, Repetition-Token, Hallucination, Semantic Coverage-Step, Commonsense, and Missing Step metrics from VCR-Bench while we design driving-specific metrics as an extension to access the reasoning steps in light of safety-critical considerations. Table [II] shows the details of these metrics. We use GPT-4o to score the generated steps and answers with reference to the ground truth using a structured evaluation prompt that includes a detailed scoring rubric, predefined rating thresholds, and a standardized JSON output format for consistency. An overall score averages over all the metrics to report a final reasoning score. Additionally, we compute the accuracy of the final answer for all multiple choice question (MCQ) based questions.

Citation

@misc{ishaq2025drivelmmo1,
  title={DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding},
  author={Ishaq et al. (2025)},
  year={2025},
  note={arXiv:2503.10621}
}

• arXiv: 2503.10621