adu2021 Watch and Learn: Learning to Use Computers from Online Videos
Core Concept
Rather than directly predicting actions from visual observations, an inverse dynamics model (IDM) infers which action likely caused the transition between consecutive video frames. This formulation simplifies trajectory extraction from web-scale video sources, enabling generation of 53,000+ executable UI demonstrations for training computer-using agents.
Architecture Overview
- Inverse Dynamics Model: Predicts action from (screen_t, screen_t+1) pairs rather than direct action regression
- Vision Backbone: SigLIP-2 Base encoder producing 1024 visual tokens per 1000×1000 screenshot
- Multi-Head Prediction: Separate branches for action classification (6 primitives), coordinate bins (1000×1000 discretization), and text generation (GPT-2 decoder)
- Video Filtering Pipeline: Automated screencast detection and quality filtering (avg frame score >0.8)
- Action Space Composition: Atomic primitives (click, release, scroll, type, wait, move) compose into environment-specific commands via deterministic mapping
Implementation Steps
1. Training the Inverse Dynamics Model
Collect 600K synthetic (screen_t, action, screen_t+1) transitions from automated web interaction using Playwright. Train a multi-task model to predict action types, coordinates, and text.
import torch
from transformers import AutoModel
class InverseDynamicsModel(torch.nn.Module):
def __init__(self):
# Vision encoder: SigLIP-2 Base (1024 tokens per screen)
self.vision_encoder = AutoModel.from_pretrained("siglip-so400m-patch14-384")
# Backbone: 4 Transformer layers
self.backbone = torch.nn.TransformerEncoder(
torch.nn.TransformerEncoderLayer(
d_model=768, nhead=12, dim_feedforward=3072, batch_first=True
),
num_layers=4
)
# Prediction heads
self.action_classifier = torch.nn.Linear(768, 6) # click, release, scroll, type, wait, move
self.coord_predictor = torch.nn.Linear(768, 2000) # 1000 bins x/y
self.text_decoder = torch.nn.LSTM(768, 768, batch_first=True)
def forward(self, screen_t, screen_t_plus_1):
# Encode both screens
tokens_t = self.vision_encoder(screen_t)
tokens_next = self.vision_encoder(screen_t_plus_1)
# Concatenate and process
combined = torch.cat([tokens_t, tokens_next], dim=1)
features = self.backbone(combined)
# Predict action, coordinates, text
action_logits = self.action_classifier(features[:, 0])
coord_logits = self.coord_predictor(features[:, 0])
text_tokens = self.text_decoder(features)
return action_logits, coord_logits, text_tokens
Training details: AdamW (lr=3e-4), batch size 256, 15 epochs on 8×A100 GPUs with bfloat16 mixed precision. Multi-task loss combines classification, coordinate regression, and language modeling with equal weighting.
2. Collecting Video Data at Scale
Query YouTube API for tutorial videos across 69 applications spanning 7 categories (productivity, programming, design, audio, utilities, system, media). Use Gemini 2.5 Flash to generate specific task-oriented search queries from initial screenshots.
def retrieve_and_filter_videos(task_instruction, initial_screen):
# Generate specific query from task and screenshot
query = gemini.generate(
f"Generate a specific YouTube search query for: {task_instruction}\n"
f"Initial screen: {initial_screen}"
)
# Retrieve candidate videos
videos = youtube_search_api.search(query, max_results=50)
# Automatic quality filtering: remove non-screencast segments
filtered = []
for video in videos:
frames = extract_frames(video, fps=1) # 1 frame/sec
# Score frames for screencast quality (model-based classifier)
scores = screencast_classifier(frames)
# Retain video if avg score > 0.8 (remove tutorials with talking heads, logos)
if scores.mean() > 0.8:
filtered.append(video)
return filtered
Video pool spans 69 applications. Curated searches ensure coverage across diverse task categories rather than biasing toward common applications.
3. Extracting Trajectories via Inverse Dynamics
For each quality-filtered video, extract frames at 1 fps and apply the trained IDM to every consecutive frame pair to generate executable action sequences.
def extract_trajectory_from_video(video_path, idm_model):
# Frame extraction
frames = extract_frames(video_path, fps=1) # O_0, O_1, ...
trajectory = [frames[0]] # Initial observation
# Apply IDM to each consecutive pair
actions = []
for t in range(len(frames) - 1):
frame_t = frames[t]
frame_t_plus_1 = frames[t + 1]
# IDM predicts action
action_type, coords, text = idm_model(frame_t, frame_t_plus_1)
# Parse predictions
action_class = action_type.argmax()
x_bin, y_bin = coords.argmax(dim=1)
x, y = x_bin / 1000, y_bin / 1000 # Denormalize
# Generate environment command (deterministic mapping)
command = compose_action(action_class, x, y, text)
actions.append(command)
trajectory.append(frame_t_plus_1)
return {
'observations': trajectory,
'actions': actions,
'num_steps': len(actions)
}
# Compose primitives into environment-compatible commands
def compose_action(action_type, x, y, text):
if action_type == 0: # Click
return f"click({x}, {y})"
elif action_type == 1: # Release (drag end)
return f"release({x}, {y})"
elif action_type == 2: # Scroll
return f"scroll({x}, {y}, {'up' if y < 0.5 else 'down'})"
elif action_type == 3: # Type
return f"type('{text}')"
elif action_type == 4: # Wait
return "wait(1)"
elif action_type == 5: # Move
return f"move({x}, {y})"
IDM achieves 95.8% action type accuracy and 91.7% overall action accuracy on test set. 53K trajectories generated across Windows, macOS, Ubuntu.
4. Using Trajectories for Agent Training
Deploy extracted trajectories in two modalities: in-context learning (format as examples) or supervised fine-tuning (direct policy training).
def in_context_learning_mode(task_instruction, trajectories):
# Retrieve relevant trajectories via semantic search
relevant_trajs = retrieve_similar_trajectories(task_instruction, trajectories)
# Format as in-context exemplars (observation, action, reasoning)
exemplars = []
for traj in relevant_trajs[:3]: # Use top 3
exemplars.append({
'observation': traj['observations'][0],
'action_sequence': traj['actions'],
'reasoning': f"Completed task in {len(traj['actions'])} steps"
})
# Prepend to agent prompt without modifying downstream logic
prompt = format_exemplars(exemplars) + "\n" + task_instruction
return prompt
def supervised_fine_tuning_mode(model, trajectories, lr=1e-5, epochs=3):
# Standard sequence modeling: predict action_t from observation_t
optimizer = torch.optim.AdamW(model.parameters(), lr=lr)
for epoch in range(epochs):
for traj in trajectories:
observations = traj['observations']
actions = traj['actions']
# Forward pass: predict actions from observations
action_logits = model.predict_action(observations[:-1])
# Loss: cross-entropy on action tokens
loss = cross_entropy(action_logits, actions)
optimizer.zero_grad()
loss.backward()
optimizer.step()
return model
Performance results: In-context mode provides +3.0 to +11.1 point improvements on OSWorld/WindowsAgentArena benchmarks. SFT on 53K trajectories achieves 24.0% success on WindowsAgentArena with 7B model under 15-step constraint (SOTA for scale).
Practical Guidance
Video Quality Matters: Automatic screencast detection filters out tutorials with excessive talking, logos, or visual noise. Threshold (avg frame score >0.8) removes ~60% of candidates but ensures trajectory quality.
Coordinate Discretization: Quantize screen coordinates into 1000 uniform bins per axis (convert regression to classification). This simplifies training and improves stability compared to continuous coordinate prediction.
Scale Advantages: Largest improvements occur in domains with abundant tutorial content (GIMP: 60K+ tutorials, VLC: 40K+). Domains with sparse tutorials (niche apps) show minimal gains.
Inference Trade-off: In-context learning avoids model updates but requires storage/retrieval of exemplars. SFT permanently encodes knowledge but requires compute-intensive training.
When to Use / When NOT to Use
Use When:
- Training computer-using agents across diverse applications
- Tutorial videos exist for target domains (productivity, design, programming software)
- You need multimodal agent training data at scale (1000s of diverse examples)
- Deployment favors semantic similarity to real human workflows
NOT For:
- Tasks with no tutorial content available (niche, proprietary applications)
- Safety-critical systems where observed human behavior may include errors
- Low-bandwidth scenarios (53K trajectories require substantial storage)
- Domains where synthetic data generation is feasible and cost-effective
Reference
This skill extracts findings from "Watch and Learn: Learning to Use Computers from Online Videos" (arXiv:2510.04673, CVPR 2026). The inverse dynamics formulation and large-scale trajectory extraction pipeline enable practical multimodal agent training from real-world video sources.