Rise Robot

RISE: Self-Improving Robot Policy with Compositional World Model

Problem Context

Vision-Language-Action (VLA) foundation models excel at semantic understanding but struggle with precise contact-rich manipulation requiring fine motor control. Physical trial-and-error RL is expensive and impractical for real robots. Standard world models struggle to balance action controllability with realistic prediction. The core challenge: learn manipulation policies efficiently without costly physical interaction.

Core Concept

RISE shifts robot learning from the physical world to imagination using a compositional world model with specialized dynamics and value components. Rather than joint training, the approach separates:

1. Dynamics Model: Predicts future observations given actions (what happens)
2. Value Model: Evaluates imagined states for progress signals (is this good)

This separation enables on-policy RL in imagination, where the policy iteratively improves by proposing actions, simulating consequences, evaluating outcomes, and updating based on advantage signals—all without touching a physical robot.

Architecture Overview

• Dynamics Model: Video diffusion architecture with Task-Centric Batching for action diversity
• Value Model: Dual-objective learning combining progress regression and Temporal-Difference signals
• Advantage Computation: Cumulative improvement across imagined trajectories
• Policy Warm-up: Offline initialization on real demonstrations
• Self-Improving Loop: Imagination-based RL for continuous refinement
• Multi-view Setup: Observations from multiple camera perspectives for robustness

Implementation

The compositional world model operates through two specialized components trained with different objectives:

class DynamicsModel(nn.Module):
    """Predicts future observations from current state and action."""

    def __init__(self):
        # Video diffusion backbone
        self.backbone = VideoDiffusion()

    def forward(self, observations, actions):
        """
        Predict next multi-view observations.
        observations: [B, C, H, W] current state
        actions: [B, action_dim] proposed action
        Returns: [B, C, H, W] predicted next observations
        """
        # Concatenate observations and action
        state_action = torch.cat([observations, actions], dim=1)
        # Denoise over iterative steps to refine prediction
        predictions = self.backbone.sample(state_action, steps=50)
        return predictions

The value model combines progress and failure signals:

class ValueModel(nn.Module):
    """Evaluates imagined trajectories for advantage computation."""

    def __init__(self, vla_backbone):
        self.backbone = vla_backbone  # Pre-trained VLA
        self.progress_head = nn.Linear(hidden_dim, 1)
        self.td_head = nn.Linear(hidden_dim, 1)

    def forward(self, trajectory):
        """
        Evaluate trajectory quality with dual objectives.
        trajectory: sequence of observations
        Returns: advantage signals for policy optimization
        """
        features = self.backbone.encode(trajectory)

        # Progress regression: how much closer to goal
        progress = self.progress_head(features)

        # Temporal-Difference learning: detect failures vs successes
        td_values = self.td_head(features)

        # Combine signals
        return progress + td_values

Self-improving policy optimization loop:

def self_improving_loop(policy, dynamics_model, value_model,
                       goal_spec, num_iterations=100):
    """
    Imagination-based RL loop without physical interaction.
    """
    for iteration in range(num_iterations):
        # Sample task from curriculum
        task = sample_task(goal_spec)

        # Generate action proposals with advantage conditioning
        optimal_advantages = compute_target_advantages(task)
        proposed_actions = policy.sample(
            observations=task.initial_obs,
            target_advantages=optimal_advantages
        )

        # Simulate in learned dynamics model
        imagined_trajectory = []
        obs = task.initial_obs
        for t in range(horizon):
            action = proposed_actions[t]
            obs = dynamics_model(obs, action)
            imagined_trajectory.append(obs)

        # Evaluate imagined trajectory
        values = value_model(imagined_trajectory)

        # Compute advantage: difference between proposed and evaluated
        advantages = optimal_advantages - values

        # Policy optimization step
        policy.update(task.initial_obs, proposed_actions, advantages)

        yield {
            'iteration': iteration,
            'mean_advantage': advantages.mean().item(),
            'policy_loss': policy.loss.item()
        }

Practical Guidance

When to use: RISE is most effective for:

• Manipulation tasks requiring precise contact dynamics
• Scenarios where physical interaction is expensive or unsafe
• Multi-step reasoning about object interactions
• Learning from VLA foundation models

Key implementation steps:

1. Data Collection: Gather 1000+ real demonstrations across manipulation scenarios
2. Dynamics Training: Train video diffusion model with task-centric batching to maximize action diversity
3. Value Model Initialization: Initialize from pre-trained VLA to leverage semantic understanding
4. Policy Warm-up: Offline RL on demonstrations to learn baseline behavior
5. Self-Improvement: Launch imagination loop with curriculum progression

Task-centric batching strategy:

• Within each batch, vary action diversity within same task
• Prevents mode collapse where dynamics model learns averaging behavior
• Prioritizes different action consequences over scenario diversity

Expected improvements:

• Dynamic brick sorting: +35% absolute performance
• Backpack packing: +45% absolute performance
• Box closing: +35% absolute performance
• Typical 3-5x faster convergence compared to physical RL

Failure modes to avoid:

• Insufficient action diversity in dynamics training → learned averaging
• Value model overfitting on initial demonstrations → poor generalization
• Too-aggressive policy updates → divergence in imagination
• Horizon too short → myopic behavior

Reference

Compositional world models enable efficient robot learning by separating dynamics prediction and value estimation. The self-improving loop demonstrates how imagination-based RL can substantially improve policies without expensive physical trial-and-error on real robots.