Probing Multimodal Llms As World Models For Driving Arxiv 2405 05956v2

Probing Multimodal LLMs as World Models for Driving

One-line decision

Use this skill when you need a benchmark testing VLMs as world models for autonomous driving understanding. Avoid it when driving is not your domain.

Skill metadata

• Skill type: driving-world-model-benchmark
• Paper kind: operational-method
• Actionability: medium
• Evidence quality: full_paper

Goal

Create a benchmark probing VLMs as world models for driving scenarios, testing understanding of physics, traffic rules, and spatial reasoning.

Problem signature

• Modality: driving scenario images with world model reasoning questions.
• Data state: curated driving scenarios with physics and reasoning questions.
• Scale regime: evaluation benchmark for driving VLMs.
• Model requirement: Any VLM for evaluation.

Use when

• You evaluate VLMs for driving understanding.
• You want to test world model capabilities.
• Driving reasoning evaluation is needed.

Do not use when

• Driving is not your domain.
• General VLM evaluation is sufficient.
• You need general world model testing.

Required inputs

• driving_scenarios: Driving scene images with scenarios.
• reasoning_questions: Questions about physics, rules, and spatial reasoning.
• ground_truth: Correct reasoning answers.

Optional inputs

• difficulty_levels: Per-question difficulty.

Outputs

• driving_benchmark: Driving world model evaluation results.
• reasoning_analysis: Analysis of VLM driving reasoning.

Assumptions and prerequisites

• VLMs should understand driving physics and rules.
• World model capabilities are testable through reasoning questions.
• Driving understanding requires spatial and temporal reasoning.

Procedure

1. Design driving scenarios Action: Create diverse driving scenarios. Why: Scenarios test different aspects of driving understanding. Note: See paper for details.
2. Create reasoning questions Action: Write questions testing physics, rules, and spatial understanding. Why: Questions probe world model capabilities. Note: See paper for details.
3. Evaluate VLMs Action: Test VLMs on the benchmark. Why: Measures driving world model capability. Note: See paper for details.
4. Analyze reasoning Action: Identify which reasoning types VLMs handle well. Why: Reveals specific strengths and weaknesses. Note: See paper for details.

Parameters to set

• scenario_types — Role: Types of driving scenarios. How to set: Include various traffic situations. Default/range: Diverse. Effect: More types test broader understanding.

Validation checks

• VLMs should demonstrate driving physics understanding.
• Questions should test genuine world model reasoning.
• Results should differentiate model capabilities.

Failure modes

• Some scenarios may be ambiguous.
• VLMs may use shortcuts rather than reasoning.
• The benchmark may not cover all driving situations.

Adaptation notes for VLM training

• Use for evaluating driving-specific VLM training data.
• Driving world model testing extends to other physical domains.
• The benchmark guides driving-specific data curation.

Implementation notes

• Include diverse driving scenarios.
• Test on leading VLMs.
• Compare to human driver reasoning.

Evidence from the paper

• The benchmark tests VLMs as world models for driving.
• Current VLMs show significant gaps in driving reasoning.
• Physics and spatial reasoning are challenging for VLMs.
• The evaluation guides driving-specific VLM improvement.

Source paper

• Title: Probing Multimodal LLMs as World Models for Driving
• Year: 2024
• Venue: arXiv
• Paper ID: arxiv-2405.05956v2
• URL: http://arxiv.org/abs/2405.05956v2
• arXiv ID: 2405.05956v2