Gorl Generative Online Rl

Summary

GoRL introduces Generative Online Reinforcement Learning, which addresses instability of expressive generative policies in online RL through structural decoupling. The method separates policy optimization into a learnable latent encoder optimized via standard RL (PPO/GRPO) and a conditional generative decoder refined on improved rollouts, using a two-timescale alternating schedule.

Core Technique

Structural Decoupling: Rather than treating the entire generative policy as a single learnable object, decompose it:

1. Latent Encoder π_θ(ε|s): Maps state to latent sample, optimized via standard RL
2. Conditional Decoder g_φ(s,ε): Generates actions from latent samples, refined on good trajectories

This avoids backpropagating through complex sampling chains, which causes gradient instability.

Two-Timescale Alternating: Use alternating optimization:

Phase 1: Optimize encoder via standard RL (PPO)
Phase 2: Refine decoder on trajectories with good returns
Phase 3: Reset encoder, keep improved decoder
Repeat

Gaussian Prior Anchoring: During decoder refinement, anchor the latent prior to a fixed N(0,I) to prevent distribution shift.

Implementation

Encoder-decoder architecture:

class GoRL_Policy:
    def __init__(self):
        self.encoder = mlp(state_dim -> latent_dim)  # π_θ
        self.decoder = mlp((state_dim + latent_dim) -> action_dim)  # g_φ

    def sample_action(self, state):
        epsilon = randn(latent_dim)  # Sample from N(0,I)
        action = self.decoder(concat(state, epsilon))
        return action

Phase 1 - Encoder RL optimization:

# Standard policy gradient with encoder
def encoder_loss(states, actions, returns):
    for state, action, ret in zip(states, actions, returns):
        epsilon, log_prob = self.encoder.sample_and_log_prob(state)
        loss = -log_prob * (ret - baseline(state))
    return loss

encoder_optimizer.step(encoder_loss())

Phase 2 - Decoder refinement:

# Collect good trajectories from current policy
good_trajectories = rollout(epsilon=0.1)  # Only keep good episodes

# Refine decoder to match good trajectories
for state, action in good_trajectories:
    # Sample latent to match action
    epsilon = invert_decoder(state, action)  # Approximate inverse
    decoder_loss = mse(decoder(state, epsilon), action)
    decoder_optimizer.step(decoder_loss)

Two-timescale schedule:

for iteration in range(num_iterations):
    # Phase 1: RL on encoder (multiple steps)
    for _ in range(encoder_updates_per_phase):
        encoder_optimizer.step(encoder_loss())

    # Phase 2: Refine decoder (multiple steps)
    for _ in range(decoder_updates_per_phase):
        decoder_optimizer.step(decoder_loss())

    # Reset encoder, keep improved decoder
    encoder = reset_to_init(encoder)

When to Use

• Online RL with complex, high-dimensional action spaces
• Scenarios where expressive generative policies improve sample efficiency
• Applications combining standard RL algorithms with generative models
• Tasks where direct gradient through sampling causes instability

When NOT to Use

• Simple action spaces where deterministic policies suffice
• Scenarios without access to rollout trajectories for decoder refinement
• Real-time RL where alternating optimization overhead is prohibitive
• Applications preferring single unified policy architecture

Key References

• Generative models for policy representation
• Policy gradient and actor-critic methods
• Online reinforcement learning and sample efficiency
• Two-timescale optimization and alternating algorithms