Relic Video World Model

Summary

RELIC is a unified framework for interactive video world modeling combining three innovations: memory-efficient spatial memory using highly compressed historical latents in KV cache, extended teacher training enabling 20-second sequence generation, and replayed back-propagation for memory-efficient distillation. Together these enable real-time 16 FPS video generation with precise action control and long-horizon consistency.

Core Technique

Compressed Spatial Memory: Store historical latents with camera pose information:

memory = [(latent_t1, camera_pose_t1), (latent_t2, camera_pose_t2), ...]

Compress latents via quantization and efficient encodings, achieving 4× compression.

Extended Teacher Training: Fine-tune base video model to generate longer sequences (20 seconds vs. 5 seconds), providing stronger supervision for long-range consistency learning.

Replayed Back-Propagation: Instead of computing gradients over full 20-second sequences (memory prohibitive), use block-wise differentiation:

# Full sequence: backward through all 400 frames
# Blocked: backward through 20-frame blocks independently

Implementation

Spatial memory structure:

class SpatialMemory:
    def __init__(self, max_history=100):
        self.latents = []  # Compressed feature maps
        self.camera_poses = []  # Camera extrinsics
        self.cache_size = 0

    def add(self, latent, pose):
        # Compress latent via quantization
        compressed = quantize(latent, bits=8)
        self.latents.append(compressed)
        self.camera_poses.append(pose)

    def retrieve(self, current_pose, k=10):
        # Find most relevant past frames
        scores = similarity(current_pose, self.camera_poses)
        indices = topk(scores, k)
        return self.latents[indices]

Extended teacher training:

# Fine-tune base model for longer sequences
teacher_model = base_video_model
for epoch in range(num_epochs):
    # Generate 20-second sequences (vs. original 5)
    video = teacher_model.generate(num_frames=480)  # 20 sec at 24fps
    loss = mse(video, ground_truth_video)
    loss.backward()
    optimizer.step()

Replayed back-propagation distillation:

def block_wise_backward(sequence, block_size=20):
    total_loss = 0
    for i in range(0, len(sequence), block_size):
        block = sequence[i:i+block_size]
        # Gradient computation only within block
        block_output = student_model(block)
        block_loss = mse(block_output, teacher_output[i:i+block_size])
        block_loss.backward()  # Only within this block
        total_loss += block_loss.detach()
    return total_loss

When to Use

• Interactive video generation for robotics simulation
• Real-time video world models requiring long-horizon consistency
• Applications needing memory-efficient video generation
• Tasks where precise action control is important

When NOT to Use

• Scenarios where short sequences (5 seconds) are sufficient
• Real-time applications where memory overhead is critical
• Tasks not requiring spatial consistency
• Offline video generation where distillation is unnecessary

Key References

• World models and video prediction
• Spatial memory and attention mechanisms
• Knowledge distillation and block-wise backpropagation
• Action-conditioned video generation