adu2021 Component ID
Spatial Token Generation Pipeline for Perception-Enhanced VLMs
Motivation
Vision-language models excel at semantic understanding but struggle with fine-grained spatial grounding and geometric reasoning. When responding to spatial questions or referring expressions, models generate text without explicitly reasoning about 2D layout or 3D structure, leading to hallucinations about object locations and spatial relationships.
The Modification
Perceptio enforces a structured output sequence that grounds language generation in explicit spatial reasoning:
Output Sequence Constraint: Segmentation tokens → Depth tokens → Text answer
This forces the model to emit spatial information before generating language, anchoring subsequent text responses in measurable 2D and 3D cues.
Perception Enhancement Mechanism
The model operates through three parallel pathways:
1. Visual Encoding Pathway - Standard vision backbone for semantic understanding (object identification, scene comprehension)
2. Segmentation Pathway - Frozen SAM2 encoder providing pixel-level mask-aware representations. Generates discretized segmentation tokens indicating spatial regions and object boundaries.
3. Depth Pathway - Pre-trained monocular depth estimator with VQ-VAE codebook that discretizes continuous depth into compact token sequences. Represents 3D geometry in discrete tokens.
These pathways run in parallel with different architectural specializations, then combine outputs.
How It Works
Token Generation Order:
- Segmentation generation phase - Model emits segmentation tokens representing spatial layout and object masks
- Depth generation phase - Model emits depth tokens encoding 3D geometry
- Language generation phase - Model generates text answer conditioned on both spatial understanding tokens
This ordering ensures spatial reasoning precedes and grounds linguistic output.
Integration with Autoregressive Decoding - The segmentation and depth tokens are inserted into the standard autoregressive generation sequence, making the approach compatible with existing VLM training and decoding pipelines.
Ablation Results
The paper demonstrates substantial gains on spatial tasks:
- RefCOCO/+/g benchmarks: State-of-the-art performance on referring expression comprehension
- HardBLINK spatial reasoning: +10.3% improvement on challenging spatial QA tasks
- VQA performance: Maintains competitive performance on general visual question-answering while strengthening spatial grounding
- Consistent improvements across diverse spatial understanding benchmarks
Conditions
- Requires pre-trained depth estimation model (monocular depth predictor)
- Requires SAM2 or similar segmentation encoder for mask representation
- Works best when tasks demand explicit spatial reasoning (referring expressions, spatial QA)
- VQ-VAE discretization requires appropriate codebook size for depth representation
Drop-In Checklist
- Integrate frozen SAM2 encoder for segmentation pathway
- Add pre-trained monocular depth estimator with VQ-VAE codebook
- Implement segmentation token generation (discretize mask outputs)
- Implement depth token generation (discretize depth via VQ-VAE)
- Enforce output sequence constraint (seg → depth → text)
- Modify autoregressive decoding to include spatial tokens
- Test on RefCOCO/+/g referring expression benchmarks
- Validate spatial reasoning improvements on HardBLINK and similar tasks
- Benchmark general VQA to ensure no regression on non-spatial tasks