Paper Rob Bpp

paper_rob__bpp

Big Picture Policies (BPP) conditions robot policies on VLM-detected behaviorally salient keyframes instead of raw observation histories. By projecting diverse rollout histories onto a compact set of task-relevant events, BPP reduces distribution shift between training and deployment, achieving 70% higher real-world success rates than the best baseline.

Paper Info

Problem & Motivation

Many manipulation tasks require memory of past events (e.g., which drawer was already opened, which mug was picked up). Naive approaches:

• Current-observation-only: Fails on tasks needing history (14.4% avg success)
• Full history conditioning: Creates spurious correlations -- policies latch onto incidental features of training histories that don't generalize to out-of-distribution trajectories at deployment (12.8% avg success)

BPP's insight: condition on a minimal, semantically meaningful subset of history frames (keyframes) detected by a VLM, dramatically reducing the space of possible inputs the policy must handle.

Method Overview

                              VLM (Gemini 3 Pro)
                              "Has a drawer just been opened?"
                                       │
                                       ▼
Wrist camera stream ──> 1 Hz query ──> Binary classifier ──> Rising-edge detector
   (50 Hz)                                                         │
                                                                   ▼
                                                        Keyframe set K_t
                                                        (latency-masked: Δ=3s)
                                                                   │
                                                                   ▼
                              ┌─────────────────────────────────────┐
                              │     Diffusion Transformer Policy    │
                              │                                     │
                              │  Inputs:                            │
                              │   • Current obs o_t (4 cameras)     │
                              │   • Keyframe obs {o_k} for k ∈ K_t │
                              │   • Proprioceptive state            │
                              │                                     │
                              │  Output:                            │
                              │   • 50-step action chunk            │
                              │     (joint positions + grippers)    │
                              └─────────────────────────────────────┘

Keyframe Detection

1. VLM query: Gemini 3 Pro receives wrist camera image at 1 Hz with a task-specific binary prompt (e.g., "has the robot just picked up a mug?")
2. Rising-edge detection: Keyframe registered when VLM output transitions from NO to YES:

   K = {t : φ(o_t) = 1 ∧ φ(o_{t-1}) = 0}
   

3. Latency masking: During training, keyframes newer than Δ=3 seconds are excluded to simulate real-world VLM latency (3-5s per query):

   K_t^Δ = {k ∈ K_t : k ≤ t - Δ}
   

Policy Architecture

The policy conditions on: π_θ(a_t | o_t, {o_k}_{k ∈ K_t^Δ})

VLM Prompts (Examples)

Robot Platform

ALOHA 2 (bimanual):

Tasks & Data

Real-World Tasks (ALOHA 2)

Simulation Tasks (MuJoCo)

Data collected from 1-5 human teleoperators to introduce behavioral diversity.

Key Hyperparameters

Results

Real-World Success Rates

BPP achieves 70% relative improvement over the best baseline (PTP) on average.

Key Ablations

Integration Notes

Reimplementation Guide

Since no code is released, here's how to reimplement BPP:

1. Keyframe detector (can be independent module):

# Pseudocode for VLM keyframe detection
class KeyframeDetector:
    def __init__(self, vlm, prompt, query_hz=1.0, latency_mask=3.0):
        self.vlm = vlm          # e.g., Gemini 3 Pro API
        self.prompt = prompt     # task-specific binary question
        self.query_hz = query_hz
        self.latency_mask = latency_mask
        self.prev_response = False
        self.keyframes = []

    def update(self, obs_t, t):
        response = self.vlm.query(obs_t["wrist_image"], self.prompt)  # binary
        # Rising-edge detection
        if response and not self.prev_response:
            self.keyframes.append(t)
        self.prev_response = response
        # Return latency-masked keyframes
        return [k for k in self.keyframes if k <= t - self.latency_mask]

2. Policy architecture: Standard Diffusion Transformer with action chunking. Can build on existing codebases:

• Diffusion Policy (Chi et al., 2023)
• ACT (for ALOHA platform)

3. Key modifications to standard diffusion policy:

• Accept variable number of keyframe observations as additional context
• ResNet34 encoder with separate weights per camera, shared across timesteps
• Concatenate/cross-attend keyframe tokens with current observation tokens

Combining with Other Methods

BPP's keyframe detection is policy-agnostic -- it can be applied to any history-conditioned policy:

• Replace naive history with VLM-detected keyframes in any transformer-based policy
• Compatible with ACT, Diffusion Policy, or VLA-based approaches
• The VLM component runs independently at 1 Hz, decoupled from the 50 Hz control loop

Tips & Gotchas

• Latency masking is critical: Without it, training assumes instant keyframe detection but deployment has 3-5s VLM latency, causing distribution shift.
• Rising-edge detection matters: Raw binary VLM output would give continuous "YES" after an event; rising-edge captures the moment of change.
• Task-specific prompts required: Each task needs a custom prompt describing the behaviorally salient event. Prompts focus on state transitions ("has just picked up"), not static states.
• VLM cost: Gemini 3 Pro at 1 Hz is feasible for real-time robotics but adds API cost and latency. Consider local VLMs for deployment.
• Data diversity helps: Using multiple teleoperators creates naturally diverse histories, which BPP handles better than naive conditioning.
• BPP is most valuable when: (a) tasks require history, (b) diverse demonstrations exist, and (c) full history conditioning fails due to distribution shift.