adu2021 Overview
Wan-Move introduces a framework for controlling motion in video generation models through latent trajectory guidance. Rather than coarse-grained motion guidance, the method enables precise control using dense point trajectories propagated through latent space without requiring architectural modifications to base models.
When to Use
- Video generation with precise object motion control
- Integrating motion control into existing image-to-video models
- Scenarios requiring dense, fine-grained trajectory specification
- Applications avoiding additional motion encoder overhead
- Need for diverse, complex motion patterns beyond standard guidance
When NOT to Use
- Simple global camera movement (simpler approaches sufficient)
- Models where architectural modification is acceptable
- Scenarios not requiring dense point-level control
- Real-time generation with strict latency requirements
Core Technique
Latent space trajectory propagation for motion-aware guidance:
# Motion-controllable video generation via latent trajectories
class WanMoveController:
def __init__(self, base_video_model):
"""
Wrap existing video generation model with motion control.
No architectural modifications to base model.
"""
self.video_model = base_video_model
self.vae = base_video_model.vae # Access VAE for latent space
def parse_dense_point_trajectories(self, motion_specification):
"""
Dense point trajectory representation for granular scene control.
Objects represented as sets of point trajectories rather than
coarse bounding boxes or simple vectors.
"""
trajectories = []
for obj in motion_specification['objects']:
# Each object gets multiple tracking points
# e.g., 16-64 points distributed across object
point_trajectory = {
'object_id': obj['id'],
'start_positions': obj['start_points'], # [num_points, 2]
'end_positions': obj['end_points'], # [num_points, 2]
'num_frames': motion_specification['num_frames']
}
# Interpolate trajectory through time
full_trajectory = self.interpolate_point_trajectory(
point_trajectory
)
trajectories.append(full_trajectory)
return trajectories
def interpolate_point_trajectory(self, point_trajectory):
"""
Temporal interpolation of point positions.
"""
start = point_trajectory['start_positions']
end = point_trajectory['end_positions']
num_frames = point_trajectory['num_frames']
# Linear interpolation through frames
timesteps = torch.linspace(0, 1, num_frames)
full_trajectory = []
for t in timesteps:
frame_positions = (1 - t) * start + t * end
full_trajectory.append(frame_positions)
return torch.stack(full_trajectory) # [num_frames, num_points, 2]
def project_trajectories_to_latent_space(self, trajectories, first_frame_image):
"""
Project point trajectories into latent space for conditioning.
Leverages VAE's learned latent representations.
"""
# Encode first frame to get latent features
first_latent = self.vae.encode(first_frame_image)
# Map trajectory points to latent space coordinates
latent_trajectories = []
for trajectory in trajectories:
# Trajectory shape: [num_frames, num_points, 2]
latent_traj = []
for frame_idx, frame_points in enumerate(trajectory):
# Project 2D pixel coordinates to latent space
latent_points = self.project_pixels_to_latent(
frame_points,
first_latent,
self.video_model.latent_scale
)
latent_traj.append(latent_points)
latent_trajectories.append(torch.stack(latent_traj))
return latent_trajectories
def propagate_latent_features_along_trajectories(self, first_frame_latent, latent_trajectories):
"""
Core insight: propagate first-frame latent features along trajectories
to create spatiotemporal guidance maps.
"""
batch_size, latent_h, latent_w, latent_c = first_frame_latent.shape
num_frames = latent_trajectories[0].shape[0]
# Initialize spatiotemporal feature map
guidance_map = torch.zeros(
batch_size,
num_frames,
latent_h,
latent_w,
latent_c
)
# For each point trajectory
for traj_idx, latent_traj in enumerate(latent_trajectories):
# Extract features from first frame at starting positions
start_positions = latent_traj[0] # [num_points, 2]
for point_idx, pos in enumerate(start_positions):
# Bilinear interpolate feature at position
h, w = int(pos[0]), int(pos[1])
h = torch.clamp(h, 0, latent_h - 1)
w = torch.clamp(w, 0, latent_w - 1)
start_feature = first_frame_latent[:, h, w, :]
# Propagate this feature along trajectory
for frame_idx, frame_pos in enumerate(latent_traj):
pos = frame_pos[point_idx]
h, w = int(pos[0]), int(pos[1])
h = torch.clamp(h, 0, latent_h - 1)
w = torch.clamp(w, 0, latent_w - 1)
# Add (blend) feature at trajectory position
guidance_map[:, frame_idx, h, w, :] += start_feature
# Normalize to reasonable range
guidance_map = guidance_map / (len(latent_trajectories) + 1e-8)
return guidance_map
def generate_with_trajectory_guidance(self, prompt, first_frame, trajectories):
"""
Generate video conditioned on trajectory guidance.
Seamless integration with existing models.
"""
# Parse trajectories
parsed_trajectories = self.parse_dense_point_trajectories(trajectories)
# Project to latent space
first_latent = self.vae.encode(first_frame)
latent_trajectories = self.project_trajectories_to_latent_space(
parsed_trajectories,
first_frame
)
# Create guidance map
guidance_map = self.propagate_latent_features_along_trajectories(
first_latent,
latent_trajectories
)
# Concatenate guidance with model input
# No architectural changes needed: just additional conditioning
conditioned_input = torch.cat([
first_latent,
guidance_map
], dim=-1)
# Generate video using base model
generated_latent = self.video_model.generate(
prompt=prompt,
initial_latent=conditioned_input,
num_frames=guidance_map.shape[1]
)
# Decode to pixel space
generated_video = self.vae.decode(generated_latent)
return generated_video
def create_movebench_evaluation(self, videos):
"""
MoveBench: new evaluation dataset for motion control.
Features diverse content, longer videos, high-quality annotations.
"""
benchmark_data = {
'videos': videos,
'annotations': {
'object_trajectories': [],
'motion_complexity': [],
'semantic_relevance': []
}
}
return benchmark_data
The framework maintains scalability through standard training procedures, integrating trajectory guidance without auxiliary modules.
Key Results
- Fine-grained motion control without architecture changes
- Integration with existing image-to-video models (Wan-I2V-14B)
- Dense point trajectory representation enables precise control
- Eliminates need for separate motion encoding modules
- MoveBench benchmark for comprehensive evaluation
Implementation Notes
- Dense point trajectories capture complex motion patterns
- Latent space propagation avoids pixel-space operations
- Feature propagation creates spatiotemporal guidance
- Seamless integration as conditioning mechanism
- No model retraining required for existing architectures
References
- Original paper: https://arxiv.org/abs/2512.08765
- Focus: Motion control in video generation
- Domain: Video synthesis, generative models