# World Models Video Prediction Robotics > Build unified world models using flow-based video generation architecture that handles Text2World, Image2World, and Video2World in a single model, trained on 200M video clips with RL post-training for improved instruction-following and video quality. - Skill: `adu2021/world-models-video-prediction-robotics` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/world-models-video-prediction-robotics` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/world-models-video-prediction-robotics/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/world-models-video-prediction-robotics --- # Title: Generate Realistic Video Worlds From Multiple Input Modalities Robotics and embodied AI require understanding how the physical world evolves. Cosmos-Predict is a unified world model that generates videos from text descriptions, images, or previous video clips using a flow-based architecture. The model operates at 1280×720 resolution, handles variable-length sequences, and grounds generation in physical reality through alignment with a physics-aware vision-language model. The key innovation is unifying multiple input modalities (text, image, video) into a single generative framework, enabling flexible control of world simulation. ## Core Concept **Unified Multimodal World Simulation**: - **Single Flow-Based Model**: Text2World, Image2World, Video2World share architecture - **Conditional Generation**: Visual or textual conditions ground video generation - **Physics-Aware Grounding**: Cosmos-Reason vision-language model provides semantic understanding - **Large-Scale Training**: 200M curated video clips for diverse world dynamics - **RL Post-Training**: Policy optimization for instruction-following and quality metrics ## Architecture Overview - **Flow-Based Generator**: Diffusion-style denoising for video generation - **Multimodal Conditioning**: Text encoders (T5/LLM) + image encoders (ViT) + video encoders - **Vision-Language Grounding**: Cosmos-Reason model for semantic grounding and physical understanding - **Resolution**: 1280×720 with variable frame counts (typically 16-128 frames) - **Training Scale**: 200M video clips, multi-GPU pre-training, RL refinement ## Implementation Steps **1. Design Flow-Based Video Generator** Implement the core generative model using flow matching. ```python class CosmosFlowVideoGenerator(nn.Module): def __init__(self, hidden_dim=2048, num_blocks=24): self.hidden_dim = hidden_dim # Video encoder/decoder using 3D convolutions self.video_encoder = nn.Sequential( nn.Conv3d(3, 64, kernel_size=(3, 8, 8), stride=(1, 4, 4), padding=(1, 3, 3)), nn.ReLU(), nn.Conv3d(64, 128, kernel_size=(3, 4, 4), stride=(1, 2, 2), padding=(1, 1, 1)), nn.ReLU() ) # Transformer blocks for temporal and spatial modeling self.transformer = nn.ModuleList([ nn.TransformerEncoderLayer(d_model=hidden_dim, nhead=8, batch_first=True) for _ in range(num_blocks) ]) self.video_decoder = nn.Sequential( nn.ConvTranspose3d(128, 64, kernel_size=(3, 4, 4), stride=(1, 2, 2), padding=(1, 1, 1)), nn.ReLU(), nn.ConvTranspose3d(64, 3, kernel_size=(3, 8, 8), stride=(1, 4, 4), padding=(1, 3, 3)) ) def encode_video(self, video): # video: [batch, frames, 3, height, width] # Permute to [batch, 3, frames, height, width] for conv3d video = video.permute(0, 2, 1, 3, 4) latent = self.video_encoder(video) return latent def decode_latent(self, latent): # latent: [batch, 128, frames, h, w] video = self.video_decoder(latent) # Permute back to [batch, frames, 3, height, width] video = video.permute(0, 2, 1, 3, 4) return video def forward(self, video, noise_level=0.0): # Encode video to latent space latent = self.encode_video(video) batch, channels, frames, h, w = latent.shape # Reshape for transformer: [batch*frames*h*w, channels] latent_flat = latent.permute(0, 2, 3, 4, 1).reshape(batch * frames * h * w, channels) # Apply transformer blocks for block in self.transformer: latent_flat = block(latent_flat.unsqueeze(0)).squeeze(0) # Reshape back latent = latent_flat.reshape(batch, frames, h, w, channels).permute(0, 4, 1, 2, 3) # Decode back to video space video_recon = self.decode_latent(latent) return video_recon ``` **2. Implement Multimodal Conditioning** Support text, image, and video inputs as conditions. ```python class MultimodalConditioner(nn.Module): def __init__(self, hidden_dim=2048): # Text encoder self.text_encoder = T5EncoderModel.from_pretrained('t5-large') self.text_projection = nn.Linear(768, hidden_dim) # Image encoder self.image_encoder = timm.create_model('vit_large_patch16', pretrained=True) self.image_projection = nn.Linear(1024, hidden_dim) # Video encoder (reuse video encoder from generator) self.video_encoder = VideoEncoder() self.video_projection = nn.Linear(256, hidden_dim) def encode_text_condition(self, text): # text: list of strings tokens = self.text_encoder.tokenizer(text, padding=True, return_tensors='pt') embeddings = self.text_encoder(**tokens).last_hidden_state # Aggregate over sequence: [batch, seq_len, 768] -> [batch, hidden_dim] condition = embeddings.mean(dim=1) condition = self.text_projection(condition) return condition def encode_image_condition(self, image): # image: [batch, 3, 720, 1280] features = self.image_encoder.forward_features(image) condition = self.image_projection(features) return condition def encode_video_condition(self, video): # video: [batch, frames, 3, height, width] # Sample key frames to reduce computation key_frames = video[:, ::4, :, :, :] # Every 4th frame latents = self.video_encoder(key_frames) condition = latents.mean(dim=1) # Average over frames condition = self.video_projection(condition) return condition def forward(self, text=None, image=None, video=None): conditions = [] if text is not None: conditions.append(self.encode_text_condition(text)) if image is not None: conditions.append(self.encode_image_condition(image)) if video is not None: conditions.append(self.encode_video_condition(video)) # Combine conditions combined_condition = torch.stack(conditions).mean(dim=0) return combined_condition ``` **3. Integrate Physics-Aware Grounding (Cosmos-Reason)** Use a vision-language model to ground generation in physical understanding. ```python class PhysicsGroundedGenerator(nn.Module): def __init__(self, base_generator, vlm_model): self.generator = base_generator self.vlm = vlm_model # Cosmos-Reason model def generate_with_grounding(self, condition, num_frames=16): # Generate candidate videos generated_video = self.generator(condition) # Grade video for physical plausibility using VLM # Ask: "Does this video show physically realistic motion?" for frame_idx in range(num_frames): frame = generated_video[frame_idx] plausibility_score = self.vlm.classify( f"Is this frame physically plausible? {frame}" ) if plausibility_score < 0.5: # Regenerate this frame with stronger physics constraint # This is a heuristic; in practice, would use more sophisticated methods pass return generated_video ``` **4. Implement RL Post-Training for Quality** Optimize video quality and instruction-following through RL. ```python def rl_post_training(generator, vlm_model, training_configs, num_steps=10000): optimizer = torch.optim.Adam(generator.parameters(), lr=1e-5) for step in range(num_steps): # Sample conditions (text, image, video) conditions = training_configs.sample_batch() # Generate videos generated_videos = generator(conditions) # Evaluate video quality quality_scores = [] for video in generated_videos: # Metrics: FVD (Fréchet Video Distance), user preference, instruction alignment fvd_score = compute_fvd(video, training_configs.reference_videos) instruction_align = vlm_model.classify( f"Does this video match the instruction: {conditions}?" ) quality = 0.6 * fvd_score + 0.4 * instruction_align quality_scores.append(quality) # Policy gradient update quality_tensor = torch.tensor(quality_scores) loss = -quality_tensor.mean() # Maximize quality optimizer.zero_grad() loss.backward() optimizer.step() if step % 100 == 0: print(f"Step {step}: Quality {quality_tensor.mean():.3f}") ``` ## Practical Guidance **When to Use**: - Robotics simulation and planning - Video content generation with physical constraints - World model learning for embodied AI - Multi-modal conditional generation tasks **Hyperparameters**: - num_frames: 16-128 (longer sequences require more compute) - resolution: 720×1280 (balance between quality and speed) - num_transformer_blocks: 24 (depth for complex dynamics) - learning_rate: 1e-4 (pre-training), 1e-5 (RL tuning) **When NOT to Use**: - Real-time inference with strict latency constraints - Scenarios where explicit physics simulation is more efficient - Deterministic trajectory prediction (videos are inherently stochastic) **Pitfalls**: - **Mode collapse in generation**: Video GAN/diffusion models can collapse to mean video; use diverse training data - **Physics violations**: Generated videos may violate physics; VLM grounding helps but isn't perfect - **Computational cost**: Generating high-resolution videos is expensive; consider progressive training **Integration Point**: Use as world model for model-based RL—train robot policies in simulation, then transfer to real world. ## Reference arXiv: https://arxiv.org/abs/2511.00062