# Urbanllava Urban Multimodal Intelligence > Unified MLLM processing four urban data types simultaneously: geospatial structures, trajectory information, satellite imagery, and street-view photos. Outperforms general-purpose models on 12-task urban benchmark with 31-375% improvements. Use for urban planning, traffic analysis, location intelligence, and smart city applications requiring integrated spatial reasoning. - Skill: `adu2021/urbanllava-urban-multimodal-intelligence` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/urbanllava-urban-multimodal-intelligence` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/urbanllava-urban-multimodal-intelligence/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Productivity - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/urbanllava-urban-multimodal-intelligence --- # UrbanLLaVA: Unified Multimodal Intelligence for Urban Systems Urban environments are fundamentally multifaceted—a location is simultaneously a point on a map, a street-view scene, a trajectory endpoint, and a node in infrastructure networks. General-purpose vision-language models fail on urban tasks because they ignore this spatial and structural complexity. UrbanLLaVA introduces the first unified multimodal model designed specifically for urban data: processing structured geospatial information, trajectory information, satellite imagery, and street views in a single system. This specialization yields major improvements—31-375% over baselines—because the model learns urban-specific reasoning patterns not present in generic datasets. The insight is that urban tasks require integrated spatial reasoning: understanding how locations relate to infrastructure, how trajectories connect places, how bird's-eye and ground-level views relate. Treating these as separate modalities (image + text) misses critical structure. ## Core Concept UrbanLLaVA builds on LLaVA but introduces urban-specific components: 1. **Unified Urban Data Representation**: Four data streams (geospatial, trajectory, satellite, street-view) processed through unified architecture 2. **Multi-Stage Training**: Task alignment → knowledge learning → mixture learning, preventing conflicts between heterogeneous urban tasks 3. **Spatial Reasoning**: Fine-tuned for geographic relationships, relative positions, route understanding 4. **Cross-Modal Alignment**: Satellite-to-street-view correspondence, map-to-trajectory grounding The framework creates synthetic instruction data (UData) across three perspectives: single-location (point), trajectory (route), and global (bird's-eye). ## Architecture Overview - **Urban Instruction Dataset (UData)**: Synthetic training data across location, trajectory, and global perspectives - **Urban Training Pipeline (UTrain)**: Three-stage progressive training preventing task interference - **Urban Benchmark (UBench)**: 12-task evaluation suite spanning geospatial QA, trajectory prediction, navigation, image matching - **Modality Encoders**: Specialized handling of each urban data type - **Spatial Grounding Module**: Relates geospatial coordinates to visual content - **Trajectory Encoder**: Sequences of locations with temporal information ## Implementation Urban instruction data generation across three perspectives: ```python import torch import torch.nn as nn from typing import Dict, List, Tuple import numpy as np class UrbanDataGenerator: """ Generates synthetic urban instruction data across three perspectives: location (single point), trajectory (route), and global (satellite). """ def __init__(self, map_data, imagery_db, trajectory_db): """ Args: map_data: Geospatial database (OSM, etc) imagery_db: Satellite and street-view image database trajectory_db: Trajectory samples from real urban mobility """ self.map_data = map_data self.imagery_db = imagery_db self.trajectory_db = trajectory_db def generate_location_view_data(self, location_id: int, num_samples: int = 100): """ Location perspective: single point with multimodal context. Example: Q: "What businesses are at coordinates (39.9°N, 116.4°E)?" A: "Shopping mall and restaurants" Returns: instruction_data: List of (question, image, answer) tuples """ instruction_data = [] for _ in range(num_samples): location = self.map_data.get_location(location_id) # Retrieve multimodal data for this location geospatial_info = self.map_data.get_geospatial_features(location_id) satellite_image = self.imagery_db.get_satellite_view(location) street_views = self.imagery_db.get_street_views(location, num_views=4) # Generate questions about the location questions = [ f"Describe the area at coordinates {location['lat']}, {location['lon']}", f"What landmarks are visible here? {satellite_image}", f"What businesses operate at this location? {geospatial_info}", f"Is this area residential or commercial? {street_views}" ] answers = [ location.get('description', 'Urban area'), self._describe_landmarks(satellite_image), self._list_businesses(geospatial_info), self._classify_area_type(street_views) ] for q, a in zip(questions, answers): instruction_data.append({ 'question': q, 'images': [satellite_image] + street_views, 'geospatial': geospatial_info, 'answer': a, 'perspective': 'location' }) return instruction_data def generate_trajectory_view_data(self, trajectory_id: int, num_samples: int = 50): """ Trajectory perspective: route through multiple locations. Example: Q: "Plan a route from home to work avoiding traffic" A: [sequence of turn instructions and coordinates] Returns: instruction_data: List of trajectory-based QA samples """ instruction_data = [] for _ in range(num_samples): trajectory = self.trajectory_db.get_trajectory(trajectory_id) locations = trajectory['locations'] # Sequence of (lat, lon) # Generate trajectory-specific questions questions = [ f"Describe the route from {locations[0]} to {locations[-1]}", f"What landmarks are passed on this route?", f"Predict the next location given this trajectory pattern", f"Is this route efficient or should it be optimized?" ] # Collect multimodal data along trajectory satellite_sequence = [ self.imagery_db.get_satellite_view(loc) for loc in locations ] street_sequence = [ self.imagery_db.get_street_views(loc, num_views=1)[0] for loc in locations ] answers = [ self._describe_route(locations, satellite_sequence), self._identify_landmarks_on_route(locations), self._predict_next_location(trajectory), self._evaluate_route_efficiency(trajectory) ] for q, a in zip(questions, answers): instruction_data.append({ 'question': q, 'images': satellite_sequence + street_sequence, 'trajectory': locations, 'answer': a, 'perspective': 'trajectory' }) return instruction_data def generate_global_view_data(self, region: Dict, num_samples: int = 75): """ Global perspective: bird's-eye view of region. Example: Q: "What is the urban structure of this area? (satellite image)" A: "Downtown area with dense buildings, parks to the east, water body to the south" Returns: instruction_data: List of global-view QA samples """ instruction_data = [] for _ in range(num_samples): # Get full satellite view of region satellite = self.imagery_db.get_satellite_view(region, zoom_level='regional') # Generate global understanding questions questions = [ f"Analyze the urban structure: {satellite}", f"Identify major infrastructure (roads, water, parks): {satellite}", f"Classify zones (residential, commercial, industrial): {satellite}", f"What is the population density pattern?" ] answers = [ self._analyze_urban_structure(satellite), self._identify_infrastructure(satellite), self._classify_zones(satellite), self._estimate_density_pattern(region) ] for q, a in zip(questions, answers): instruction_data.append({ 'question': q, 'images': [satellite], 'region': region, 'answer': a, 'perspective': 'global' }) return instruction_data # Helper methods (simplified) def _describe_landmarks(self, image): return "Major buildings and structures visible" def _list_businesses(self, info): return info.get('businesses', []) def _classify_area_type(self, views): return "Mixed residential-commercial" def _describe_route(self, locs, imgs): return f"Route through {len(locs)} locations" def _identify_landmarks_on_route(self, locs): return "Notable landmarks passed" def _predict_next_location(self, traj): return f"Next location: {traj['locations'][-1]}" def _evaluate_route_efficiency(self, traj): return "Route is efficient" def _analyze_urban_structure(self, sat): return "Dense urban center with suburban fringe" def _identify_infrastructure(self, sat): return ["Major roads", "Parks", "Water bodies"] def _classify_zones(self, sat): return ["Downtown", "Residential", "Industrial"] def _estimate_density_pattern(self, region): return "High density center, decreasing outward" class UrbanLLaVA(nn.Module): """ Unified MLLM for urban tasks combining geospatial, trajectory, satellite, and street-view modalities. """ def __init__(self, model_name='llava-13b'): super().__init__() # Base LLaVA model self.llava = self._load_llava(model_name) # Urban-specific modules self.geospatial_encoder = GeospatialEncoder() self.trajectory_encoder = TrajectoryEncoder() self.satellite_encoder = SatelliteEncoder() self.street_encoder = StreetViewEncoder() # Spatial fusion module self.spatial_fusion = SpatialFusionModule() # Urban reasoning head self.urban_reasoning_head = nn.Sequential( nn.Linear(4 * 768, 1024), nn.GELU(), nn.Linear(1024, 768) ) def forward( self, question: str, satellite_images: torch.Tensor = None, street_images: torch.Tensor = None, geospatial_data: Dict = None, trajectory_data: List[Tuple] = None ) -> str: """ Process urban question across multiple modalities. Args: question: Natural language query satellite_images: (B, C, H, W) satellite view(s) street_images: (B, C, H, W) street view(s) geospatial_data: {"buildings": [...], "roads": [...], ...} trajectory_data: [[(lat, lon), ...], ...] sequences Returns: answer: Natural language response """ # Encode each modality modality_features = [] if satellite_images is not None: sat_feat = self.satellite_encoder(satellite_images) # (B, 768) modality_features.append(sat_feat) if street_images is not None: street_feat = self.street_encoder(street_images) # (B, 768) modality_features.append(street_feat) if geospatial_data is not None: geo_feat = self.geospatial_encoder(geospatial_data) # (B, 768) modality_features.append(geo_feat) if trajectory_data is not None: traj_feat = self.trajectory_encoder(trajectory_data) # (B, 768) modality_features.append(traj_feat) # Fuse modalities spatially if modality_features: fused = torch.cat(modality_features, dim=-1) # (B, 4*768) urban_context = self.urban_reasoning_head(fused) # (B, 768) else: urban_context = torch.zeros(1, 768) # Feed to LLaVA with urban context answer = self.llava.generate( question, image_context=urban_context, max_length=256 ) return answer def _load_llava(self, model_name): """Load pretrained LLaVA model.""" from transformers import AutoModelForCausalLM return AutoModelForCausalLM.from_pretrained(model_name) class UrbanTrainingPipeline: """ Three-stage training preventing task interference. """ def __init__(self, model, train_data): self.model = model self.train_data = train_data self.optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4) def stage1_task_alignment(self, num_epochs: int = 5): """ Stage 1: Align model to understand urban task diversity. Train on examples from all three perspectives without optimization. """ print("Stage 1: Task Alignment") for epoch in range(num_epochs): # Sample balanced batches from all perspectives batch = self._sample_balanced_batch() for sample in batch: loss = self._compute_loss(sample) self.optimizer.zero_grad() loss.backward() self.optimizer.step() def stage2_knowledge_learning(self, num_epochs: int = 20): """ Stage 2: Learn domain-specific knowledge for urban tasks. Focus on accuracy improvement. """ print("Stage 2: Knowledge Learning") for epoch in range(num_epochs): # Sample harder negatives, optimize accuracy batch = self._sample_challenging_batch() for sample in batch: # Compute loss with harder supervision loss = self._compute_knowledge_loss(sample) self.optimizer.zero_grad() loss.backward() torch.nn.utils.clip_grad_norm_(self.model.parameters(), 1.0) self.optimizer.step() def stage3_mixture_learning(self, num_epochs: int = 10): """ Stage 3: Learn mixture of skills across tasks. Joint optimization prevents catastrophic forgetting. """ print("Stage 3: Mixture Learning") for epoch in range(num_epochs): # Sample from all tasks proportionally batch = self._sample_mixed_batch() for sample in batch: loss = self._compute_loss(sample) self.optimizer.zero_grad() loss.backward() self.optimizer.step() def _sample_balanced_batch(self): return [] # Implementation def _sample_challenging_batch(self): return [] def _sample_mixed_batch(self): return [] def _compute_loss(self, sample): return torch.tensor(0.0) def _compute_knowledge_loss(self, sample): return torch.tensor(0.0) class GeospatialEncoder(nn.Module): """Encodes structured geospatial data (buildings, roads, POIs).""" def forward(self, data): return torch.randn(1, 768) class TrajectoryEncoder(nn.Module): """Encodes sequences of locations with temporal info.""" def forward(self, data): return torch.randn(1, 768) class SatelliteEncoder(nn.Module): """Encodes bird's-eye satellite imagery.""" def forward(self, images): return torch.randn(images.shape[0], 768) class StreetViewEncoder(nn.Module): """Encodes street-level panoramic images.""" def forward(self, images): return torch.randn(images.shape[0], 768) class SpatialFusionModule(nn.Module): """Fuses multimodal urban data spatially.""" def forward(self, features): return features ``` ## Practical Guidance | Aspect | Value | Notes | |--------|-------|-------| | Urban Task Categories | 12 diverse tasks | Geospatial QA, trajectory, navigation, matching | | Benchmark Coverage | Single/cross-modal | Comprehensive urban applications | | vs. GPT-4o Performance | +31-375% | Significant improvements on urban tasks | | Training Stages | 3 progressive | Prevents task interference | | Data Perspectives | 3 (location, trajectory, global) | Comprehensive coverage | | Model Scale | Typically 7B-13B | Efficient for urban deployment | **When to use:** - Urban planning and development analysis - Traffic and transportation optimization - Location intelligence and property assessment - Smart city applications requiring spatial reasoning - Navigation and route planning systems - Urban imagery analysis and classification - Multi-modal geospatial understanding tasks **When NOT to use:** - Non-urban geographic analysis (rural, natural features) - Tasks requiring real-time inference on edge devices (model scale) - Scenarios without multimodal urban data available - General computer vision where urban specialization unnecessary - High-security applications (model fine-tuning may leak training data) **Common pitfalls:** - Assuming urban specialization helps all location-based tasks (it's specifically urban) - Training data imbalance across three perspectives - Geospatial encoding losing coordinate precision - Not leveraging trajectory patterns (they contain rich temporal info) - Confusing satellite-level and street-level reasoning requirements - Stage training skipped, causing task interference - Evaluating on tasks outside the 12-benchmark scope ## Reference "UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understanding", 2025. [arxiv.org/abs/2506.23219](https://arxiv.org/abs/2506.23219)