adu2021 What This Skill Does
Augment pretrained video transformers with tactile sensor streams and a cross-modal regularization loss that enforces balanced attention between visual and tactile modalities, enabling robots to handle contact-rich manipulation tasks where vision alone provides incomplete state information.
The Component Swap
Old component: Video-Action Models (VAMs) that encode world state from video tokens alone, missing critical tactile cues about contact forces, pressure, and transition events.
# Traditional VAM: vision-only
class VideoActionModel(nn.Module):
def __init__(self, video_encoder, action_head):
super().__init__()
self.video_encoder = video_encoder
def forward(self, video_frames, language_instruction):
# video_frames: [B, T, 3, H, W]
video_tokens = self.video_encoder(video_frames) # [B, T*N, D]
action = self.action_head(video_tokens)
return action # Missing tactile information
New component: Multimodal fusion with explicit tactile stream and regularization to prevent visual latent dominance.
class VTAM(nn.Module):
def __init__(self, video_encoder, tactile_encoder, action_head, hidden_dim=256):
super().__init__()
self.video_encoder = video_encoder # Pretrained video transformer
self.tactile_encoder = tactile_encoder # Lightweight tactile processing
self.action_head = action_head
self.hidden_dim = hidden_dim
# Cross-modal fusion layer
self.fusion = nn.MultiheadAttention(
embed_dim=hidden_dim,
num_heads=8,
batch_first=True
)
# Tactile regularization loss weight
self.tactile_loss_weight = 0.1
def forward(self, video_frames, tactile_data, language_instruction):
"""
video_frames: [B, T, 3, H, W]
tactile_data: [B, T, 6] - 3 force axes + 3 torque axes per timestep
language_instruction: [B, L] - tokenized instruction
"""
# Encode both modalities
video_tokens = self.video_encoder(video_frames) # [B, T*N_v, D]
tactile_tokens = self.tactile_encoder(tactile_data) # [B, T, D]
# Expand tactile to match video token count (repeat across spatial tokens)
# This ensures tactile info is available for every spatial region
tactile_tokens_expanded = tactile_tokens.unsqueeze(2).expand(
-1, -1, video_tokens.shape[1] // tactile_data.shape[1], -1
).reshape(tactile_tokens.shape[0], -1, self.hidden_dim)
# Multimodal fusion with cross-attention
video_query = video_tokens
tactile_key_value = tactile_tokens_expanded
fused, attn_weights = self.fusion(
video_query,
tactile_key_value,
tactile_key_value
)
# Predict action
action = self.action_head(fused)
return action, attn_weights
def compute_tactile_regularization_loss(self, attn_weights):
"""
Ensure tactile modality receives sufficient attention weight.
Prevents visual dominance that would ignore contact information.
Loss encourages: max(tactile_attn) > threshold
"""
# attn_weights: [B, T*N_v, T*N_t]
# Sum attention from each video token to all tactile tokens
tactile_attention_per_token = attn_weights.sum(dim=-1).mean()
# Loss: penalize if tactile attention is too low
# Target: at least 10-20% of total attention budget to tactile
target_tactile_attn = 0.15
loss_tactile_reg = (target_tactile_attn - tactile_attention_per_token) ** 2
return loss_tactile_reg
def training_loss(self, action_pred, action_true, attn_weights):
"""Total loss combines action prediction and tactile regularization."""
loss_action = F.mse_loss(action_pred, action_true)
loss_tactile = self.compute_tactile_regularization_loss(attn_weights)
total_loss = loss_action + self.tactile_loss_weight * loss_tactile
return total_loss
The key swap is from single-modality (video-only) to dual-stream fusion with a regularization term that prevents the model from ignoring tactile information, even when visual cues are available. This forces the model to learn to interpret contact states that vision cannot represent.
Performance Impact
Contact-rich manipulation tasks:
- Average success rate: 90% (substantial improvement from baseline VAM)
- Contact state detection: significantly improved
Potato chip pick-and-place (high-precision, high-force task):
- Improvement over π0.5 baseline: 80% relative improvement
- Baseline struggles with fragile object requiring precise force control
- VTAM's tactile feedback enables appropriate gripper modulation
Task complexity correlation:
- Simple reach tasks: minimal tactile benefit
- Object placement: moderate benefit
- Contact-sensitive manipulation (chip, egg, deformable): substantial benefit (20-80% improvement)
When to Use
- Contact-rich manipulation tasks (object insertion, pushing, deformable objects)
- Scenarios where visual observability is partial (gripper occludes object)
- When force control is critical (assembly, delicate object handling)
- Tasks with contact state transitions (initial contact → grasp → lift)
- Robotic systems equipped with tactile sensors (F/T wrist sensor, tactile skin)
When NOT to Use
- Purely visual tasks where contact is incidental (reaching without contact)
- If tactile sensors are unavailable or unreliable
- On systems with latency constraints (tactile encoding adds compute)
- When tactile data is noisy and unreliable for your sensors
- Very simple tasks where vision alone is sufficient (overhead not justified)
Implementation Checklist
1. Tactile encoder setup:
- Ensure tactile data pipeline: 6-axis F/T sensor (or equivalent) → normalized to [-1, 1]
- Synchronize tactile and video timestamps (often tactile runs at higher frequency; downsample or aggregate)
- Create lightweight tactile encoder: 2-3 layer MLP or 1D conv
class TactileEncoder(nn.Module):
def __init__(self, input_size=6, hidden_size=256, output_size=256):
super().__init__()
self.net = nn.Sequential(
nn.Linear(input_size, hidden_size),
nn.ReLU(),
nn.Linear(hidden_size, output_size)
)
def forward(self, tactile_data):
# tactile_data: [B, T, 6] → [B, T, output_size]
return self.net(tactile_data)
2. Architecture integration:
# Minimal swap: add tactile branch to existing VAM
vtam = VTAM(
video_encoder=your_pretrained_video_encoder,
tactile_encoder=TactileEncoder(input_size=6),
action_head=your_action_head,
hidden_dim=256
)
# Forward pass now accepts tactile data
action = vtam(video_frames, tactile_data, instruction)
3. Fine-tuning strategy:
- Freeze pretrained video encoder weights (or use LoRA for light fine-tuning)
- Train tactile encoder and fusion layer on your task
- Typical: 10K-50K steps on task-specific data
4. Verification:
- Measure task success rate on contact-sensitive tasks
- Compare: VAM baseline (vision only) vs. VTAM
- Ablate: remove tactile regularization loss to confirm it's necessary
- Check attention weights: verify tactile tokens receive >10% of attention
5. Hyperparameter tuning if needed:
- Tactile loss weight: start at 0.1; increase to 0.5 if visual still dominates
- Target tactile attention: 0.15 (15% of budget); adjust based on task criticality
- Tactile encoder hidden size: match video encoder dimension (e.g., 256)
- Learning rate: 1e-4 for frozen video encoder, 1e-3 for trainable encoder
6. Known issues:
- Tactile sensor noise: add moving average filter or Kalman filter for smoothing
- Synchronization drift: drift between video and tactile timestamps causes misalignment
- Limited contact data: if contact-rich trajectories are rare, use data augmentation (simulate contacts)
- Generalization: model trained on one robot/sensor may not transfer; validate on your hardware
- Cold start: if tactile info is absent early in episode, model may learn to ignore it; ensure diverse training trajectories
Related Work
This builds on multimodal fusion architectures (CLIP, BLIP) and extends VAM-style video prediction with sensor fusion patterns. The tactile regularization loss parallels auxiliary task learning (MTL) and relates to balanced contrastive objectives in multimodal transformers. Inspired by human sensorimotor control, where tactile feedback is essential for object manipulation.