adu2021 Voyager: Long-Range and World-Consistent Video Diffusion
Core Concept
Voyager addresses a critical limitation in 3D scene generation: most methods cannot simultaneously generate RGB and depth sequences that remain spatially consistent across arbitrarily long videos. By jointly generating aligned RGB-depth pairs and implementing efficient world caching with point culling, Voyager enables direct 3D reconstruction without expensive post-processing pipelines.
Architecture Overview
- Geometry-Injected Conditioning: Use partial RGB and depth maps rather than RGB-only conditioning to reduce hallucinations in complex occlusions
- Depth-Fused Diffusion: Concatenate RGB and depth along spatial dimensions for pixel-level interaction
- World Caching System: Accumulate 3D points from generated frames; point culling removes redundancy, reducing memory ~40% while maintaining essential geometry
- Auto-Regressive Sampling: Generate arbitrary-length videos while maintaining world consistency through incremental point cloud updates
- Camera Path Control: Accept user-specified camera trajectories for controlled scene exploration
Implementation
Step 1: Prepare Dual-Modal Training Dataset
from typing import List, Dict, Tuple
import numpy as np
class DualModalDatasetBuilder:
def __init__(self, num_training_pairs=100000):
self.target_size = num_training_pairs
self.auto_annotate_pipeline = self.AnnotationPipeline()
class AnnotationPipeline:
def estimate_metric_depth(self, single_image):
"""Use pretrained depth estimator (MiDaS, etc) to get metric depth"""
depth_estimator = load_pretrained_depth_model('midas')
relative_depth = depth_estimator(single_image)
# Convert to metric scale using known scene structures
metric_depth = self.scale_to_metric(relative_depth)
return metric_depth
def generate_camera_poses(self, image, num_frames=30):
"""Automatically generate camera trajectories"""
poses = []
# Infer scene bounds from depth and image content
scene_bounds = self.estimate_scene_bounds(image)
# Generate smooth camera path (circular, spiral, etc)
for t in np.linspace(0, 1, num_frames):
pose = self.generate_smooth_camera_motion(
scene_bounds, t
)
poses.append(pose)
return poses
def render_target_frames(self, single_image, camera_poses):
"""Render RGB-D video from single image using 3D warping"""
# Use depth map to create 3D point cloud from image
point_cloud = self.image_to_point_cloud(
single_image,
self.estimate_metric_depth(single_image)
)
rgb_frames = []
depth_frames = []
for pose in camera_poses:
# Render point cloud from camera pose
rgb_frame = self.render_rgb(point_cloud, pose)
depth_frame = self.render_depth(point_cloud, pose)
rgb_frames.append(rgb_frame)
depth_frames.append(depth_frame)
return rgb_frames, depth_frames
def build_training_dataset(self, single_images: List):
"""Create training pairs: single image -> RGB-D video"""
dataset = []
for img in single_images:
# Auto-annotate with depth
depth_map = self.auto_annotate_pipeline.estimate_metric_depth(img)
# Generate camera poses
camera_poses = self.auto_annotate_pipeline.generate_camera_poses(
img, num_frames=30
)
# Render target RGB-D sequence
rgb_frames, depth_frames = (
self.auto_annotate_pipeline.render_target_frames(
img, camera_poses
)
)
# Create training example
example = {
'input_image': img,
'input_depth': depth_map,
'camera_poses': camera_poses,
'target_rgb_frames': rgb_frames,
'target_depth_frames': depth_frames,
'scene_id': len(dataset),
}
dataset.append(example)
print(f"Built dataset: {len(dataset)} single-image -> video pairs")
print(f"Total frames: {len(dataset) * 30}")
return dataset
# Build 100K+ training pairs
builder = DualModalDatasetBuilder(num_training_pairs=100000)
training_data = builder.build_training_dataset(images)
Step 2: Implement Depth-Fused Diffusion Model
import torch
import torch.nn as nn
class DepthFusedDiffusionModel(nn.Module):
def __init__(self, num_channels=8, num_layers=12):
super().__init__()
# 8 channels: 3 RGB + 1 Depth + 4 extra features
self.num_channels = num_channels
# Transformer blocks for spatial interaction
self.spatial_fusion = nn.ModuleList([
self.SpatialFusionBlock(num_channels)
for _ in range(num_layers)
])
# Temporal consistency module
self.temporal_consistency = self.TemporalModule()
class SpatialFusionBlock(nn.Module):
"""Enable pixel-level interaction between RGB and depth"""
def __init__(self, channels):
super().__init__()
# Cross-modal attention: RGB queries, depth keys/values
self.rgb_to_depth_attn = nn.MultiheadAttention(
embed_dim=channels, num_heads=8
)
self.depth_to_rgb_attn = nn.MultiheadAttention(
embed_dim=channels, num_heads=8
)
def forward(self, rgb, depth):
"""
Args:
rgb: [B, 3, H, W] - RGB channels
depth: [B, 1, H, W] - Depth channel
Returns:
fused: [B, 4, H, W] - Interacted RGB-D
"""
B, _, H, W = rgb.shape
# Flatten spatial dimensions for attention
rgb_flat = rgb.view(B, 3, -1).transpose(1, 2) # [B, HW, 3]
depth_flat = depth.view(B, 1, -1).transpose(1, 2) # [B, HW, 1]
# Cross-attention: let depth inform RGB
rgb_informed, _ = self.depth_to_rgb_attn(
rgb_flat, depth_flat, depth_flat
)
# Cross-attention: let RGB inform depth
depth_informed, _ = self.rgb_to_depth_attn(
depth_flat, rgb_flat, rgb_flat
)
# Reshape back to spatial
rgb_informed = rgb_informed.transpose(1, 2).view(B, 3, H, W)
depth_informed = depth_informed.transpose(1, 2).view(B, 1, H, W)
# Concatenate fused representations
fused = torch.cat([rgb_informed, depth_informed], dim=1)
return fused
class TemporalModule(nn.Module):
"""Ensure consistency across video frames"""
def __init__(self):
super().__init__()
self.temporal_attention = nn.MultiheadAttention(
embed_dim=4, num_heads=4
)
def forward(self, frame_sequence):
"""
Args:
frame_sequence: [T, B, 4, H, W] - T frames of RGB-D
Returns:
consistent_frames: [T, B, 4, H, W]
"""
T, B, C, H, W = frame_sequence.shape
# Compute temporal attention to smooth transitions
frames_flat = frame_sequence.view(T, B*H*W, C)
attended, _ = self.temporal_attention(
frames_flat, frames_flat, frames_flat
)
return attended.view(T, B, C, H, W)
def forward(self, image, depth_init, camera_poses, num_frames=30):
"""
Generate RGB-D video from single image and camera poses.
Args:
image: [B, 3, H, W] - Input RGB image
depth_init: [B, 1, H, W] - Estimated depth of input image
camera_poses: [B, num_frames, 4, 4] - Camera transformation matrices
num_frames: int - Number of frames to generate
"""
# Diffusion forward pass for first frame
rgb_frames = []
depth_frames = []
# Use input as conditioning
rgb_cond = image
depth_cond = depth_init
for frame_idx in range(num_frames):
# Camera transformation: warp previous frame using depth
if frame_idx > 0:
warped_rgb = self.warp_frame(
rgb_frames[-1],
depth_frames[-1],
camera_poses[:, frame_idx]
)
else:
warped_rgb = rgb_cond
# Denoise RGB-D jointly
noisy_frame = self.add_noise(
torch.cat([warped_rgb, depth_cond], dim=1),
noise_level=frame_idx / num_frames # Less noise as we progress
)
# Apply spatial fusion blocks
denoised = noisy_frame
for fusion_block in self.spatial_fusion:
rgb_part = denoised[:, :3, :, :]
depth_part = denoised[:, 3:4, :, :]
denoised = fusion_block(rgb_part, depth_part)
# Split into RGB and depth
rgb_frame = denoised[:, :3, :, :]
depth_frame = denoised[:, 3:4, :, :]
rgb_frames.append(rgb_frame)
depth_frames.append(depth_frame)
# Apply temporal consistency
frame_sequence = torch.stack(rgb_frames + depth_frames, dim=0)
consistent = self.temporal_consistency(frame_sequence)
return rgb_frames, depth_frames
def warp_frame(self, rgb, depth, camera_pose):
"""Warp frame using depth and camera transformation"""
# Implement backward warping using depth and pose
return rgb # Simplified placeholder
Step 3: Implement World Caching System
import open3d as o3d
from collections import defaultdict
class WorldCachingSystem:
def __init__(self, memory_reduction_target=0.6):
self.point_cloud = o3d.geometry.PointCloud()
self.frame_points = defaultdict(list)
self.memory_reduction_target = memory_reduction_target
def accumulate_points_from_frame(self, rgb_frame, depth_frame,
camera_pose, intrinsics):
"""Add 3D points from new frame to world cache"""
# Project depth + camera pose to get 3D points
points_3d = self.depth_to_world_coordinates(
depth_frame, camera_pose, intrinsics
)
# Get colors from RGB frame
colors = self.sample_colors_from_rgb(rgb_frame, points_3d)
# Add to accumulating point cloud
new_points = o3d.geometry.PointCloud()
new_points.points = o3d.utility.Vector3dVector(points_3d)
new_points.colors = o3d.utility.Vector3dVector(colors)
self.point_cloud += new_points
return len(points_3d)
def point_culling(self, culling_ratio=0.4):
"""Remove redundant points to reduce memory"""
initial_size = len(self.point_cloud.points)
# Downsample point cloud aggressively
culled = self.point_cloud.voxel_down_sample(
voxel_size=0.05 # Adjust based on scene scale
)
# Use statistical outlier removal
culled, _ = culled.remove_statistical_outlier(
nb_neighbors=20,
std_ratio=2.0
)
final_size = len(culled.points)
actual_reduction = 1.0 - (final_size / initial_size)
print(f"Point culling: {initial_size} -> {final_size} points")
print(f"Memory reduction: {actual_reduction:.1%}")
self.point_cloud = culled
return actual_reduction
def auto_regressive_generation(self, model, image, depth_init,
camera_poses_all, batch_size=8):
"""
Generate video auto-regressively, periodically calling point culling.
This enables generating arbitrary-length videos without memory explosion.
"""
rgb_frames = []
depth_frames = []
culling_interval = 30 # Cull points every N frames
for frame_batch_idx in range(0, len(camera_poses_all), batch_size):
batch_poses = camera_poses_all[
frame_batch_idx : frame_batch_idx + batch_size
]
# Generate batch of frames
rgb_batch, depth_batch = model(
image, depth_init, batch_poses
)
rgb_frames.extend(rgb_batch)
depth_frames.extend(depth_batch)
# Accumulate points
for rgb_f, depth_f, pose in zip(rgb_batch, depth_batch, batch_poses):
self.accumulate_points_from_frame(
rgb_f, depth_f, pose, intrinsics
)
# Periodically cull redundant points
if len(rgb_frames) % culling_interval == 0:
reduction = self.point_culling(culling_ratio=0.4)
print(f"Frame {len(rgb_frames)}: Memory reduction {reduction:.1%}")
# Final output
return rgb_frames, depth_frames, self.point_cloud
Practical Guidance
Geometry-Aware Conditioning: Always provide depth alongside RGB conditioning. Depth prevents the diffusion model from hallucinating surfaces in occluded regions, dramatically improving consistency.
Joint RGB-Depth Generation: Design diffusion models to generate both modalities simultaneously with cross-attention, not as separate pipelines. Pixel-level RGB-depth interaction is crucial.
Camera Path Specification: Support user-defined camera trajectories (circular, spiral, free-form). This gives users control and enables reproducible generation.
World Caching for Infinite Video: Don't generate monolithic long videos. Instead, accumulate points, periodically cull redundancy (removing ~40% without visible quality loss), and continue generating. This enables arbitrarily long videos.
Point Culling Parameters: Use voxel downsampling with small voxel sizes (0.05 unit scale) and statistical outlier removal (20 neighbors, 2.0 std ratio). Tune based on target scene density.
Evaluation: Assess both visual quality (RGB fidelity) and geometric consistency (3D reconstruction error). A good model excels at both.
Reference
- Paper: Voyager (2506.04225)
- Architecture: Depth-fused diffusion transformer with world caching
- Dataset: 100,000+ automatic RGB-D video pairs via single-image rendering
- Key Innovation: Joint RGB-depth generation with efficient world state management