Drivelm Driving With Graph Visual Question Answering Arxiv 2312 14150v2

DriveLM: Driving with Graph Visual Question Answering

One-line decision

Use this skill when you want to create structured QA data for autonomous driving VLMs using graph-based scene understanding. Avoid it when you do not work with autonomous driving or need general-purpose VQA.

Skill metadata

• Skill type: driving-qa-data-generation
• Paper kind: operational-method
• Actionability: high
• Evidence quality: full_paper

Goal

Create structured visual QA data for autonomous driving by building graph representations of driving scenes and generating questions about perception, prediction, planning, and behavior.

Problem signature

• Modality: driving scene images/videos with structured QA about driving tasks.
• Data state: driving scene data annotated with graph-based QA covering perception, prediction, and planning.
• Scale regime: thousands of driving scenes with comprehensive QA.
• Model requirement: VLM for driving scene understanding; graph annotation pipeline.

Use when

• You need driving-specific visual QA data.
• You want structured reasoning about driving scenes.
• You need perception, prediction, and planning QA.

Do not use when

• You do not work with autonomous driving.
• General VQA is sufficient for your needs.
• You lack driving scene data.

Required inputs

• driving_scenes: Driving scene images/videos from nuScenes or similar.
• graph_annotations: Graph representations of scene objects and relationships.
• qa_templates: Templates for perception, prediction, planning, and behavior questions.

Optional inputs

• 3d_annotations: 3D bounding box and trajectory annotations.

Outputs

• drivelm_qa_data: Structured driving QA dataset.
• driving_vlm: VLM fine-tuned for driving scene understanding.

Assumptions and prerequisites

• Graph-based scene representation enables systematic QA generation.
• Driving tasks can be decomposed into perception, prediction, and planning.
• Structured QA teaches reasoning about driving scenarios.

Procedure

1. Build scene graphs Action: Create graph representations of driving scenes with objects, relationships, and actions. Why: Graphs provide structured information for QA generation. Note: See paper for details.
2. Generate perception questions Action: Create questions about what objects are present, their attributes, and locations. Why: Perception is the foundation of driving understanding. Note: See paper for details.
3. Generate prediction questions Action: Create questions about what objects will do next. Why: Prediction is critical for safe driving. Note: See paper for details.
4. Generate planning questions Action: Create questions about appropriate driving actions. Why: Planning tests end-to-end driving reasoning. Note: See paper for details.
5. Train driving VLM Action: Fine-tune a VLM on the generated driving QA data. Why: Specialized training enables driving scene understanding. Note: See paper for details.

Parameters to set

• qa_categories — Role: Categories of driving questions. How to set: Include perception, prediction, planning, behavior. Default/range: 4 categories. Effect: Comprehensive coverage tests all driving reasoning aspects.
• scene_complexity — Role: Complexity of driving scenarios. How to set: Include simple and complex traffic scenarios. Default/range: Diverse. Effect: Complex scenarios test deeper reasoning.

Validation checks

• Perception questions should test correct object identification.
• Prediction questions should have verifiable future states.
• Planning questions should test safe driving decisions.

Failure modes

• Graph annotations may miss important scene elements.
• Template-based QA may lack diversity.
• Driving reasoning may be too specialized for general VLMs.

Adaptation notes for VLM training

• DriveLM-style data is essential for autonomous driving VLMs.
• The graph-to-QA approach generalizes to other structured domains.
• Combine with general instruction data for balanced training.

Implementation notes

• Use nuScenes annotations as the basis for graphs.
• Generate diverse question phrasings for robustness.
• Track per-category QA accuracy.

Evidence from the paper

• DriveLM creates structured driving QA data using graph-based scene understanding.
• Questions cover perception, prediction, planning, and behavior reasoning.
• The graph structure enables systematic and comprehensive QA generation.
• DriveLM enables end-to-end driving reasoning evaluation for VLMs.

Source paper

• Title: DriveLM: Driving with Graph Visual Question Answering
• Year: 2023
• Venue: ECCV
• Paper ID: arxiv-2312.14150v2
• URL: http://arxiv.org/abs/2312.14150v2
• arXiv ID: 2312.14150v2