adu2021 G²VLM: Geometry-Grounded Vision-Language Models
Standard vision-language models treat images as 2D sequences, missing opportunities to ground reasoning in 3D spatial structure. This skill demonstrates how to augment VLMs with geometric perception by adding a specialized expert stream that reconstructs 3D properties (point clouds, normals, camera poses) in parallel with semantic understanding—enabling models to reason about spatial relationships, scale, occlusion, and layout.
The core innovation is the Mixture-of-Transformer-Experts architecture with paired geometric and semantic experts that interact through shared attention, enabling mutual improvement between visual geometry and language understanding.
Core Concept
G²VLM implements geometric grounding through:
- Geometric Perception Expert ("where" pathway): Reconstructs 3D space from 2D observations
- Semantic Perception Expert ("what" pathway): Handles multimodal understanding and reasoning
- Interaction Mechanism: Shared self-attention enabling cross-expert information flow
- Unified Decoding: Both pathways contribute to downstream VLM reasoning tasks
Architecture Overview
- Vision Encoder (shared): DINOV2 encoding of input image
- Geometric Expert: Predicts 3D coordinates, normals, camera parameters
- Semantic Expert: Processes visual+text for reasoning tasks
- Cross-Expert Attention: Experts share information through transformer layers
- Output Heads: Geometry predictions + semantic reasoning
Implementation Steps
The model architecture integrates geometric and semantic pathways from image encoding onward.
1. Build Shared Vision Encoder
Initialize encoder using pretrained vision model (DINOV2 recommended).
class SharedVisionEncoder(torch.nn.Module):
"""
Encodes images into feature representations for both geometric and semantic experts.
Uses pretrained DINOV2 for strong geometric understanding.
"""
def __init__(self, model_name='dinov2_vitb14', freeze=True):
super().__init__()
# Load pretrained DINOV2
self.encoder = torch.hub.load(
'facebookresearch/dinov2',
model_name
)
# Optionally freeze backbone
if freeze:
for param in self.encoder.parameters():
param.requires_grad = False
self.feature_dim = self.encoder.embed_dim
self.patch_size = self.encoder.patch_embed.patch_size
def forward(self, images):
"""
Encode images to patch-level features.
Args:
images: (batch, 3, height, width)
Returns:
features: (batch, num_patches, feature_dim)
patch_positions: (batch, num_patches, 2) normalized patch coordinates
"""
# Forward through encoder
features = self.encoder.forward_features(images)['x']
# Compute patch positions for 3D grounding
batch_size, num_patches, feature_dim = features.shape
# Grid of patch positions
grid_h = int(np.sqrt(num_patches))
grid_w = grid_h
patch_positions = torch.zeros(batch_size, num_patches, 2)
for i in range(grid_h):
for j in range(grid_w):
patch_idx = i * grid_w + j
patch_positions[:, patch_idx, 0] = j / grid_w
patch_positions[:, patch_idx, 1] = i / grid_h
return features, patch_positions
2. Implement Geometric Expert for 3D Prediction
Build expert stream predicting 3D properties from visual features.
class GeometricExpert(torch.nn.Module):
"""
Predicts 3D geometric properties from visual features.
Outputs: point coordinates, surface normals, camera pose.
"""
def __init__(self, feature_dim=768, num_points=1024, hidden_dim=512):
super().__init__()
self.feature_dim = feature_dim
self.num_points = num_points
self.hidden_dim = hidden_dim
# Transformer layers for geometric reasoning
self.geometry_transformer = torch.nn.TransformerEncoder(
torch.nn.TransformerEncoderLayer(
d_model=feature_dim,
nhead=8,
dim_feedforward=hidden_dim,
batch_first=True
),
num_layers=4
)
# Output heads for 3D predictions
self.point_head = torch.nn.Sequential(
torch.nn.Linear(feature_dim, hidden_dim),
torch.nn.ReLU(),
torch.nn.Linear(hidden_dim, hidden_dim // 2),
torch.nn.ReLU(),
torch.nn.Linear(hidden_dim // 2, num_points * 3) # (x, y, z) per point
)
self.normal_head = torch.nn.Sequential(
torch.nn.Linear(feature_dim, hidden_dim),
torch.nn.ReLU(),
torch.nn.Linear(hidden_dim, hidden_dim // 2),
torch.nn.ReLU(),
torch.nn.Linear(hidden_dim // 2, 3) # Surface normal
)
self.camera_head = torch.nn.Sequential(
torch.nn.Linear(feature_dim, hidden_dim),
torch.nn.ReLU(),
torch.nn.Linear(hidden_dim, 6) # Camera pose (position + rotation)
)
def forward(self, features, patch_positions=None):
"""
Predict 3D geometry from visual features.
Args:
features: (batch, num_patches, feature_dim)
patch_positions: (batch, num_patches, 2) patch coordinates
Returns:
point_map: (batch, num_points, 3) 3D point coordinates
surface_normals: (batch, 3) average surface normal
camera_pose: (batch, 6) camera position and orientation
"""
# Process features through transformer
geometric_features = self.geometry_transformer(features)
# Global feature for camera pose
global_feature = geometric_features.mean(dim=1)
# Predict 3D points
point_logits = self.point_head(geometric_features.mean(dim=1))
point_map = point_logits.reshape(-1, self.num_points, 3)
# Predict surface normals
surface_normals = self.normal_head(geometric_features.mean(dim=1))
surface_normals = torch.nn.functional.normalize(surface_normals, dim=-1)
# Predict camera pose
camera_pose = self.camera_head(global_feature)
return {
'point_map': point_map,
'surface_normals': surface_normals,
'camera_pose': camera_pose
}
3. Implement Semantic Expert for VLM Reasoning
Build semantic expert handling text and semantic visual understanding.
class SemanticExpert(torch.nn.Module):
"""
Semantic understanding pathway handling text+vision reasoning.
Inherits geometric grounding from geometric expert via shared attention.
"""
def __init__(self, feature_dim=768, text_vocab_size=30522, hidden_dim=512):
super().__init__()
self.feature_dim = feature_dim
# Text embedding
self.text_embedder = torch.nn.Embedding(text_vocab_size, feature_dim)
# Semantic transformer (interacts with geometric expert via attention)
self.semantic_transformer = torch.nn.TransformerEncoder(
torch.nn.TransformerEncoderLayer(
d_model=feature_dim,
nhead=8,
dim_feedforward=hidden_dim,
batch_first=True
),
num_layers=4
)
# VLM task heads
self.vqa_head = torch.nn.Sequential(
torch.nn.Linear(feature_dim, hidden_dim),
torch.nn.ReLU(),
torch.nn.Linear(hidden_dim, 1000) # VQA answers
)
self.caption_head = torch.nn.Sequential(
torch.nn.Linear(feature_dim, hidden_dim),
torch.nn.ReLU(),
torch.nn.Linear(hidden_dim, text_vocab_size)
)
def forward(self, visual_features, text_input_ids, geometric_features=None):
"""
Process visual+text input for semantic reasoning.
Args:
visual_features: (batch, num_patches, feature_dim) from vision encoder
text_input_ids: (batch, text_seq_len) tokenized text
geometric_features: (batch, num_patches, feature_dim) from geometric expert
Returns:
vqa_logits: (batch, num_answers)
caption_logits: (batch, text_seq_len, vocab_size)
"""
# Embed text
text_features = self.text_embedder(text_input_ids)
# Combine visual + text + geometric features
combined_features = visual_features + text_features.mean(dim=1, keepdim=True)
if geometric_features is not None:
# Incorporate geometric information
combined_features = combined_features + 0.3 * geometric_features
# Semantic reasoning
semantic_output = self.semantic_transformer(combined_features)
# Task-specific outputs
vqa_logits = self.vqa_head(semantic_output.mean(dim=1))
caption_logits = self.caption_head(semantic_output)
return {
'vqa_logits': vqa_logits,
'caption_logits': caption_logits,
'features': semantic_output
}
4. Build Mixture-of-Experts Architecture
Integrate geometric and semantic experts with shared attention.
class GeometryGroundedVLM(torch.nn.Module):
"""
Complete G²VLM model with geometric and semantic experts.
Experts interact through shared transformer layers.
"""
def __init__(self, vision_model_name='dinov2_vitb14', feature_dim=768):
super().__init__()
self.vision_encoder = SharedVisionEncoder(vision_model_name)
self.geometric_expert = GeometricExpert(feature_dim=feature_dim)
self.semantic_expert = SemanticExpert(feature_dim=feature_dim)
# Cross-expert attention layers
self.cross_attention = torch.nn.MultiheadAttention(
feature_dim, num_heads=8, batch_first=True
)
# Gate for blending expert outputs
self.expert_gate = torch.nn.Linear(feature_dim * 2, 2)
def forward(self, images, text_input_ids=None, return_geometry=False):
"""
Forward pass through G²VLM with geometric and semantic pathways.
Args:
images: (batch, 3, height, width) input images
text_input_ids: (batch, text_len) optional text input
return_geometry: Whether to return 3D predictions
Returns:
vqa_logits: (batch, num_answers)
geometry: Dict with 3D predictions (optional)
"""
# Encode image
visual_features, patch_positions = self.vision_encoder(images)
# Geometric expert forward pass
geometry = self.geometric_expert(visual_features, patch_positions)
# Semantic expert forward pass
semantic_output = self.semantic_expert(
visual_features, text_input_ids, geometry_features=None
)
# Cross-expert interaction: geometric expert attends to semantic
if text_input_ids is not None:
attended_geometric, _ = self.cross_attention(
visual_features, semantic_output['features'], semantic_output['features']
)
# Gate: blend attended geometric with original semantic
combined = torch.cat([semantic_output['features'], attended_geometric], dim=-1)
gate_logits = self.expert_gate(combined)
gate_weights = torch.softmax(gate_logits, dim=-1)
semantic_output['features'] = (
gate_weights[:, :, 0:1] * semantic_output['features'] +
gate_weights[:, :, 1:2] * attended_geometric
)
results = {
'vqa_logits': semantic_output['vqa_logits'],
'caption_logits': semantic_output['caption_logits'],
'semantic_features': semantic_output['features']
}
if return_geometry:
results['geometry'] = geometry
return results
5. Training Loss with Geometry Supervision
Define multi-task loss combining VLM and 3D prediction tasks.
def geometry_grounded_loss(
model_output,
targets,
geometry_targets=None,
lambda_geo=0.3,
lambda_vqa=1.0,
lambda_caption=0.5
):
"""
Combined loss for VLM tasks + geometry prediction.
Args:
model_output: Dict from model forward pass
targets: Dict with 'vqa_answers', 'captions'
geometry_targets: Dict with '3d_points', 'normals', 'camera_pose'
lambda_geo, lambda_vqa, lambda_caption: Loss weights
Returns:
total_loss: Combined loss for backpropagation
"""
losses = {}
# VQA loss
vqa_loss = torch.nn.functional.cross_entropy(
model_output['vqa_logits'],
targets['vqa_answers']
)
losses['vqa'] = lambda_vqa * vqa_loss
# Caption loss
caption_loss = torch.nn.functional.cross_entropy(
model_output['caption_logits'].reshape(-1, model_output['caption_logits'].shape[-1]),
targets['captions'].reshape(-1)
)
losses['caption'] = lambda_caption * caption_loss
# Geometry losses (if supervision available)
if geometry_targets is not None and 'geometry' in model_output:
geometry = model_output['geometry']
# 3D point prediction loss
point_loss = torch.nn.functional.mse_loss(
geometry['point_map'],
geometry_targets['3d_points']
)
losses['points'] = point_loss
# Surface normal prediction loss
normal_loss = torch.nn.functional.mse_loss(
geometry['surface_normals'],
geometry_targets['normals']
)
losses['normals'] = normal_loss
# Camera pose prediction loss
camera_loss = torch.nn.functional.mse_loss(
geometry['camera_pose'],
geometry_targets['camera_pose']
)
losses['camera'] = camera_loss
geo_loss = point_loss + normal_loss + camera_loss
losses['geometry'] = lambda_geo * geo_loss
total_loss = sum(losses.values())
return total_loss, losses
6. Inference with Spatial Reasoning
Use geometry-grounded model for spatially-aware VLM inference.
def geometry_aware_vqa(
model,
image,
question,
tokenizer,
use_geometry=True
):
"""
Answer VQA question using geometry-grounded reasoning.
Args:
model: Trained G²VLM
image: Input image (PIL or tensor)
question: Question string
tokenizer: Text tokenizer
use_geometry: Whether to incorporate 3D predictions
Returns:
answer: Generated answer text
geometry: Optional 3D predictions for spatial reasoning
"""
# Tokenize question
question_tokens = tokenizer(question, return_tensors='pt')['input_ids']
# Forward pass
output = model(
image,
text_input_ids=question_tokens,
return_geometry=use_geometry
)
# Decode VQA answer
answer_idx = torch.argmax(output['vqa_logits'], dim=-1)
answer = tokenizer.decode([answer_idx])
geometry = output.get('geometry', None)
return answer, geometry
Practical Guidance
When to Use G²VLM:
- Vision tasks requiring spatial understanding (scene graphs, 3D QA)
- Applications needing explicit geometry (robotics, 3D scene understanding)
- Tasks combining semantic reasoning with spatial relationships
- Models that benefit from additional 3D supervision signal
When NOT to Use:
- Pure 2D tasks (classification, 2D detection)
- Scenarios without 3D annotation data for pretraining
- Real-time applications where 3D prediction overhead is unacceptable
Key Hyperparameters:
lambda_geo: Weight of geometry loss (0.1-0.5)num_geometric_transformer_layers: Depth of geometry expert (2-6)num_3d_points: Resolution of point cloud prediction (256-2048)feature_dim: Vision feature dimension (typically 768)
Training Data Requirements:
- 3D annotations (point clouds, normals) enable better geometric learning
- Can leverage synthetic 3D data for pretraining
- Transfer from 2D-only models possible with fine-tuning (slightly lower final performance)
Computational Overhead:
- ~20-30% increased inference time vs. standard VLM (additional geometric expert)
- Memory increase: ~15-20% due to extra transformer layers
- Can optimize by sharing some transformer layers between experts
Reference
Research paper: https://arxiv.org/abs/2511.21688