Offline Rl

Offline Reinforcement Learning

When to Use This Skill

Invoke this skill when you encounter:

• Offline Dataset: User has fixed dataset D, no environment access
• Distribution Shift: Policy improvements diverge from data collection distribution
• Value Overestimation: Learning from offline data causes Q-values to diverge
• CQL/IQL Implementation: Implementing conservative value estimation
• Behavior Cloning Constraints: How to keep policy close to data distribution
• BCQ Algorithm: Batch-Constrained Q-learning for offline control
• Offline Evaluation: Evaluating policy performance without environment interaction
• When to Apply Offline RL: Deciding if offline RL is needed vs online alternatives
• Extrapolation Error: Understanding why agents fail on OOD actions
• Data Quality Impact: How dataset composition affects algorithm choice

This skill teaches learning from fixed datasets without environment interaction.

Do NOT use this skill for:

• Online RL with environment interaction (use policy-gradient-methods, actor-critic-methods)
• Pure supervised learning on (s,a) pairs (that's behavior cloning, use supervised learning)
• Online model-free learning (use value-based-methods)
• Algorithm-agnostic debugging (use rl-debugging-methodology)

Core Principle

Offline RL learns from fixed datasets without environment interaction, solving the fundamental problem of value overestimation without online correction.

The core insight: Standard RL algorithms (Q-learning, policy gradient) assume you can interact with environment to correct mistakes. Offline RL has no such luxury.

Online RL Problem:
  1. Agent explores environment
  2. Collects (s,a,r,s',d) transitions
  3. Updates value estimate: Q[s,a] ← r + γ max_a' Q[s',a']
  4. If Q overestimates, agent tries bad action
  5. Environment gives low reward
  6. Q-value corrects downward in next update

Offline RL Problem:
  1. Agent receives fixed dataset D
  2. Estimates Q from D only
  3. Updates value estimate: Q[s,a] ← r + γ max_a' Q[s',a']
  4. If Q overestimates (no data for some actions), agent tries bad action
  5. No feedback! Can't try action in environment
  6. Q-value never corrects. Error compounds.
  7. Policy diverges, performance collapses

Without understanding extrapolation error and conservative value estimation, you'll implement algorithms that hallucinate value for unseen state-action pairs.

Part 1: The Offline RL Problem

Offline RL Fundamentals

Offline RL Setting:

• You have fixed dataset D = {(s_i, a_i, r_i, s'_i, d_i)} collected by unknown behavior policy
• No access to environment (can't interact)
• Goal: Learn policy π that maximizes expected return
• Constraint: Policy must work on real environment, not just in data

Key Difference from Supervised Learning:

Supervised Learning (behavior cloning):
  π = argmin_π E_{(s,a)~D}[||π(a|s) - β(a|s)||²]
  Problem: Learns data collection policy, can't improve

Offline RL:
  π = argmin_π E_{(s,a,r,s')~D}[Q(s,a) - μ(a|s)]
  Benefit: Uses reward signal to improve beyond data
  Challenge: Q-values unreliable outside data distribution

Why Standard Q-Learning Fails on Offline Data

The Extrapolation Problem:

Imagine discrete MDP with 3 actions: left, right, wait.

# Data collection: random policy samples uniformly
# States: {s1, s2}
# Dataset D:
# (s1, left, r=5, s1')
# (s1, left, r=4, s1')
# (s1, right, r=3, s1')
# (s2, wait, r=10, s2')
# (s2, wait, r=9, s2')
# (s1, right, r=2, s1')

# Training Q-Learning on D:
Q(s1, left) ≈ 4.5   # Average of data
Q(s1, right) ≈ 2.5  # Average of data
Q(s1, wait) ≈ ???   # No data! Network must extrapolate

# What does Q-network guess for Q(s1, wait)?
# Network sees:
#   - action=left → reward high
#   - action=right → reward low
#   - action=wait → no signal
# Worst case: network predicts Q(s1, wait) = 100 (hallucination!)

# Policy improvement:
π(a|s1) = argmax_a Q(s1, a)
         = argmax{4.5, 2.5, 100}
         = wait  # WRONG CHOICE!

# In real environment: s1 + wait = crash (reward = -100)
# Why? Network extrapolated wildly beyond training distribution.

The Root Cause:

1. Training signal only for actions in D
2. Network trained to minimize MSE on seen (s,a) pairs
3. For unseen (s,a), network interpolates/extrapolates
4. Extrapolation is unreliable in high dimensions
5. Policy picks extrapolated high values
6. In reality: catastrophic failure

Distribution Shift and Policy Divergence

The Core Challenge:

Initial behavior policy β:
  Collects D by visiting state distribution d_β
  D covers: {actions β takes, rewards β gets}

Your learned policy π:
  Improves on β
  Visits different states: d_π
  Tries different actions: π(a|s)

Mismatch:
  - States in d_π but not d_β: Q has no data
  - Actions π takes but β doesn't: Q extrapolates
  - Q-estimates unreliable
  - Policy gets stuck in hallucinated high-value regions

Example: Robot Manipulation:

Data: Collected by [move_forward, move_left, grasp] policy
- Good at pushing objects forward
- Poor at pulling backward

Your offline-trained π:
- Learns forward motion works (in data)
- Learns backward motion has high Q (extrapolated!)
- Because backward unexplored, network guesses Q=50

Reality:
- Forward: actually good
- Backward: crashes into wall, r=-100
- Policy fails catastrophically

Why?
- Distribution shift: π tries backward, d_π ≠ d_β
- No data for backward actions
- Network extrapolates incorrectly

Value Overestimation: The Central Problem

Standard Q-learning uses:

Q(s,a) ← Q(s,a) + α(r + γ max_{a'} Q(s',a') - Q(s,a))
                            └─ Greedy max

The Problem with max_a':

With limited data:
- max_{a'} Q(s',a') picks highest Q-value
- But if Q overestimates for most actions
- max picks an overestimated value
- TD target is too high
- Q-values drift upward indefinitely

Why Does This Happen Offline?

Online RL:

Iteration 1:
  Q(s,a) = 10 (overestimate)
Iteration 2:
  Real transition: (s,a,r=-5,s')
  Q(s,a) ← -5 + γ max Q(s',a')
  If Q(s',a') corrected to 0, Q(s,a) ← -5
  Overestimation corrected!

Offline RL:
  Same transition in replay buffer D
  Q(s,a) ← -5 + γ max Q(s',a')
  But max Q(s',a') still overestimated (no correction)
  Q stays high, continues overestimating
  Error never corrects

Part 2: Conservative Q-Learning (CQL)

CQL: The Idea

Conservative Q-Learning directly addresses value overestimation by adding a pessimistic lower bound:

Standard Bellman (optimistic):
  Q(s,a) ← r + γ max_{a'} Q(s',a')

CQL (conservative):
  Q(s,a) ← r + γ max_{a'} (Q(s',a') - α * C(a'))

Where C(a') is a penalty for actions outside data distribution.

Key Idea: Penalize high Q-values for actions not well-represented in data.

CQL in Practice: The Implementation

Full CQL Update:

import torch
import torch.nn as nn
from torch.optim import Adam

class CQLAgent:
    def __init__(self, state_dim, action_dim, hidden_dim=256):
        # Q-network (standard DQN-style)
        self.Q = nn.Sequential(
            nn.Linear(state_dim + action_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, 1)  # Q-value output
        )

        # Behavior cloning network (estimate β(a|s))
        self.pi_b = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, action_dim)
        )

        self.Q_optimizer = Adam(self.Q.parameters(), lr=3e-4)
        self.pi_b_optimizer = Adam(self.pi_b.parameters(), lr=3e-4)

        # CQL hyperparameters
        self.cql_weight = 1.0  # How much to penalize OOD actions
        self.discount = 0.99
        self.target_update_rate = 0.005

        # Target network
        self.Q_target = nn.Sequential(
            nn.Linear(state_dim + action_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, 1)
        )
        self._soft_update_target()

    def train_step(self, batch):
        """
        One CQL training update on batch D.
        Batch contains: states, actions, rewards, next_states, dones
        """
        states, actions, rewards, next_states, dones = batch
        batch_size = states.shape[0]

        # 1. Compute TD target with CQL penalty
        with torch.no_grad():
            # Next action values
            q_next = self.Q_target(torch.cat([next_states, actions], dim=1))

            # CQL: penalize high Q-values for OOD actions
            # Sample random actions and batch actions
            random_actions = torch.rand((batch_size, 10, self.action_dim))
            batch_actions = actions.unsqueeze(1).expand(-1, 10, -1)

            # Q-values for random and batch actions
            q_random = self.Q_target(torch.cat([
                next_states.unsqueeze(1).expand(-1, 10, -1),
                random_actions
            ], dim=2))  # [batch, 10]

            q_batch = self.Q_target(torch.cat([
                next_states.unsqueeze(1).expand(-1, 10, -1),
                batch_actions
            ], dim=2))  # [batch, 10]

            # CQL penalty: log(sum exp(Q_random) + sum exp(Q_batch))
            # This penalizes taking OOD actions
            cql_penalty = (
                torch.logsumexp(q_random, dim=1) +
                torch.logsumexp(q_batch, dim=1)
            )

            # TD target: conservative value estimate
            td_target = rewards + (1 - dones) * self.discount * (
                q_next - self.cql_weight * cql_penalty / 2
            )

        # 2. Update Q-network
        q_pred = self.Q(torch.cat([states, actions], dim=1))
        q_loss = ((q_pred - td_target) ** 2).mean()

        self.Q_optimizer.zero_grad()
        q_loss.backward()
        self.Q_optimizer.step()

        # 3. Update behavior cloning network (optional)
        # Helps estimate which actions are in-distribution
        log_probs = torch.log_softmax(self.pi_b(states), dim=1)
        bc_loss = -log_probs.gather(1, actions.long()).mean()

        self.pi_b_optimizer.zero_grad()
        bc_loss.backward()
        self.pi_b_optimizer.step()

        # 4. Soft update target network
        self._soft_update_target()

        return {
            'q_loss': q_loss.item(),
            'bc_loss': bc_loss.item(),
            'cql_penalty': cql_penalty.mean().item()
        }

    def _soft_update_target(self):
        """Soft update target network toward main network."""
        for target_param, main_param in zip(
            self.Q_target.parameters(),
            self.Q.parameters()
        ):
            target_param.data.copy_(
                self.target_update_rate * main_param.data +
                (1 - self.target_update_rate) * target_param.data
            )

    def select_action(self, state, temperature=0.1):
        """
        Select action using CQL-trained Q-values.
        Temperature controls exploration.
        """
        with torch.no_grad():
            # Evaluate all actions (in discrete case)
            q_values = []
            for a in range(self.action_dim):
                action_tensor = torch.tensor([a], dtype=torch.float32)
                q_val = self.Q(torch.cat([state, action_tensor]))
                q_values.append(q_val.item())

            q_values = torch.tensor(q_values)

            # Softmax policy (temperature for uncertainty)
            logits = q_values / temperature
            action_probs = torch.softmax(logits, dim=0)

            # Sample or take greedy
            action = torch.multinomial(action_probs, 1).item()
            return action

Key CQL Components:

1. CQL Penalty: logsumexp(Q_random) + logsumexp(Q_batch)

   • Penalizes high Q-values for both random and batch actions
   • Forces Q-network to be pessimistic
   • Prevents extrapolation to unseen actions

2. Conservative Target: r + γ(Q(s',a') - α*penalty)

   • Lowers TD target by CQL penalty amount
   • Makes Q-estimates more conservative
   • Safer for policy improvement

3. Behavior Cloning Network: Estimates β(a|s)

   • Helps identify in-distribution actions
   • Can weight CQL penalty by action probability
   • Tighter constraint on constrained actions

CQL Intuition

What CQL Prevents:

Without CQL:
Q(s1, wait) = 100  (hallucinated, no data)
π picks wait → disaster

With CQL:
Q_target for s1, wait includes penalty
Q_target = r + γ(Q(s',a') - α * log(sum exp(Q)))
         = r + γ(50 - 100)
         = r - 50 * γ
CQL pessimism forces Q(s1, wait) low
π picks safer action

Result: Policy stays in data distribution, avoids hallucinated values

When CQL Works Well

• Short horizons: Errors don't compound as much
• Diverse data: Multiple actions represented
• Known behavior policy: Can weight penalty appropriately
• Discrete actions: Easier to evaluate all actions

CQL Failure Modes

• Too conservative on good data: May not improve over β
• High variance penalties: log-sum-exp can be unstable
• Computational cost: Requires sampling many actions

Part 3: Implicit Q-Learning (IQL)

IQL: A Different Approach to Pessimism

While CQL explicitly penalizes OOD actions, IQL achieves pessimism through a different mechanism: expectile regression.

Standard L2 Regression (mean):
  Expected value minimizes E[(y - ŷ)²]

Expectile Regression (quantile-like):
  Expects value minimizes E[|2τ - 1| * |y - ŷ|] for τ in (0,1)
  - τ < 0.5: underestimates (pessimistic)
  - τ = 0.5: median (neutral)
  - τ > 0.5: overestimates (optimistic)

IQL Implementation

Key Insight: Use expectile loss to make Q-estimates naturally pessimistic without explicit penalties.

class IQLAgent:
    def __init__(self, state_dim, action_dim, hidden_dim=256, expectile=0.7):
        self.Q = self._build_q_network(state_dim, action_dim, hidden_dim)
        self.V = self._build_v_network(state_dim, hidden_dim)  # Value function

        self.Q_optimizer = Adam(self.Q.parameters(), lr=3e-4)
        self.V_optimizer = Adam(self.V.parameters(), lr=3e-4)

        self.expectile = expectile  # τ = 0.7 for slight pessimism
        self.discount = 0.99
        self.temperature = 1.0  # For policy softness

    def expectile_loss(self, diff, expectile):
        """
        Asymmetric expectile loss.
        Penalizes overestimation more than underestimation (pessimism).
        """
        weight = torch.where(
            diff > 0,
            expectile * torch.ones_like(diff),
            (1 - expectile) * torch.ones_like(diff)
        )
        return weight * (diff ** 2)

    def train_v_function(self, batch):
        """
        Step 1: Train value function V(s)
        V(s) estimates expected Q-value under behavior policy
        """
        states, actions, rewards, next_states, dones = batch

        # Q-values from current policy
        q_values = self.Q(states, actions)

        # V-network predicts these Q-values
        v_pred = self.V(states)

        # Expectile loss: V should underestimate Q slightly
        # (stay pessimistic)
        q_diff = q_values - v_pred
        v_loss = self.expectile_loss(q_diff, self.expectile).mean()

        self.V_optimizer.zero_grad()
        v_loss.backward()
        self.V_optimizer.step()

        return {'v_loss': v_loss.item()}

    def train_q_function(self, batch):
        """
        Step 2: Train Q-function using pessimistic V-target
        Q(s,a) ← r + γ V(s')  (instead of γ max_a' Q(s',a'))
        """
        states, actions, rewards, next_states, dones = batch

        # IQL target: use V-function instead of max Q
        with torch.no_grad():
            v_next = self.V(next_states)
            td_target = rewards + (1 - dones) * self.discount * v_next

        q_pred = self.Q(states, actions)
        q_loss = ((q_pred - td_target) ** 2).mean()

        self.Q_optimizer.zero_grad()
        q_loss.backward()
        self.Q_optimizer.step()

        return {'q_loss': q_loss.item()}

    def train_step(self, batch):
        """IQL training: V-function first, then Q-function."""
        v_info = self.train_v_function(batch)
        q_info = self.train_q_function(batch)
        return {**v_info, **q_info}

    def select_action(self, state):
        """
        Policy improvement: use exponential weighted Q-values.
        Only improve actions with high estimated value.
        """
        with torch.no_grad():
            # Evaluate actions
            q_values = self.Q(state, actions=None)  # All actions

            # Exponential weighting: exp(Q/τ)
            # Concentrates on high-Q actions
            weights = torch.exp(q_values / self.temperature)
            weights = weights / weights.sum()

            action = torch.multinomial(weights, 1).item()
            return action

IQL Key Insight

Why V-function Makes Q Pessimistic:

Standard Q-Learning:
  Q(s,a) = r + γ max_a' Q(s',a')  ← Optimistic!

IQL:
  1. Train V(s) to estimate Q under behavior policy
     V(s) ≈ E_a~β[Q(s,a)]
  2. Use V as target: Q(s,a) = r + γ V(s')
     Why pessimistic?

     If policy is suboptimal:
       - Good actions: Q > V(s) (above average)
       - Bad actions: Q < V(s) (below average)
       - max_a Q(s',a') picks good action (might extrapolate)
       - V(s') is average (conservative)

    Result: Using V instead of max prevents picking overestimated actions!

Expectile Loss Intuition

Standard MSE: E[(Q - V)²]
  - Symmetric penalty: overestimation = underestimation

Expectile Loss (τ=0.7): E[|2*0.7 - 1| * |Q - V|²]
  - When Q > V: weight = 0.7 (moderate penalty)
  - When Q < V: weight = 0.3 (light penalty)
  - Result: V underestimates Q slightly

Effect: Q values are naturally pessimistic without explicit penalties!

When IQL Excels

• High-dimensional observations: V-function is simpler than Q
• Continuous actions: No need to discretize
• Mixed quality data: Expectile naturally handles varying data quality
• Implicit distribution shift handling: V-function implicitly constrains to data distribution

Part 4: Batch-Constrained Q-Learning (BCQ)

BCQ: Constraining Policy to Behavior Support

Core Idea: Only improve actions that have high probability under behavior policy β.

Standard offline improvement:
  π(a|s) ← exp(Q(s,a) / τ)  ← Can pick any action!

BCQ improvement:
  π(a|s) ← exp(Q(s,a) / τ) * I(β(a|s) > threshold)
           └─ Only if action has nonzero β probability

BCQ Implementation

class BCQAgent:
    def __init__(self, state_dim, action_dim, hidden_dim=256):
        # Q-network
        self.Q = self._build_q_network(state_dim, action_dim, hidden_dim)

        # Behavior cloning network: learns β(a|s)
        self.pi_b = self._build_policy_network(state_dim, action_dim, hidden_dim)

        # Perturbation network: learn action perturbations near β
        # π(a|s) = arg max_a Q(s, β(s) + ξ(s, a))
        self.perturbation = self._build_perturbation_network(
            state_dim, action_dim, hidden_dim
        )

        self.Q_optimizer = Adam(self.Q.parameters(), lr=3e-4)
        self.pi_b_optimizer = Adam(self.pi_b.parameters(), lr=3e-4)
        self.perturbation_optimizer = Adam(self.perturbation.parameters(), lr=3e-4)

        self.discount = 0.99
        self.constraint_weight = 0.5  # How strict is batch constraint?

    def train_step(self, batch):
        """BCQ training loop."""
        states, actions, rewards, next_states, dones = batch

        # 1. Train behavior cloning network β
        log_probs = torch.log_softmax(self.pi_b(states), dim=1)
        pi_b_loss = -log_probs.gather(1, actions.long()).mean()

        self.pi_b_optimizer.zero_grad()
        pi_b_loss.backward()
        self.pi_b_optimizer.step()

        # 2. Train Q-network with BCQ constraint
        with torch.no_grad():
            # Behavior actions in next state
            pi_b_next = self.pi_b(next_states)  # [batch, action_dim]

            # Perturbation network learns small deviations from β
            perturbation = self.perturbation(next_states, pi_b_next)

            # Constrained action: π(a|s') = β(s') + ξ(s')
            # But in action space [-1, 1] (clipped)
            constrained_actions = torch.clamp(
                pi_b_next + perturbation, min=-1, max=1
            )

            # Q-value with constrained action
            q_next = self.Q(next_states, constrained_actions)
            td_target = rewards + (1 - dones) * self.discount * q_next

        # Q-loss
        q_pred = self.Q(states, actions)
        q_loss = ((q_pred - td_target) ** 2).mean()

        self.Q_optimizer.zero_grad()
        q_loss.backward()
        self.Q_optimizer.step()

        # 3. Train perturbation network
        # Learn to find best perturbation to β that increases Q
        pi_b_curr = self.pi_b(states)
        perturbation_curr = self.perturbation(states, pi_b_curr)
        perturbed_actions = torch.clamp(
            pi_b_curr + perturbation_curr, min=-1, max=1
        )

        q_perturbed = self.Q(states, perturbed_actions)
        perturbation_loss = -q_perturbed.mean()  # Maximize Q

        self.perturbation_optimizer.zero_grad()
        perturbation_loss.backward()
        self.perturbation_optimizer.step()

        return {
            'q_loss': q_loss.item(),
            'pi_b_loss': pi_b_loss.item(),
            'perturbation_loss': perturbation_loss.item()
        }

    def select_action(self, state, num_samples=100):
        """
        BCQ action selection:
        1. Sample from β(a|s)
        2. Perturb by small amount
        3. Pick action with highest Q
        """
        with torch.no_grad():
            # Behavior policy action
            pi_b = self.pi_b(state)

            # Sample small perturbations
            perturbations = []
            for _ in range(num_samples):
                pert = self.perturbation(state, pi_b)
                perturbed = torch.clamp(pi_b + pert, min=-1, max=1)
                perturbations.append(perturbed)

            perturbations = torch.stack(perturbations)  # [num_samples, action_dim]

            # Evaluate Q for each perturbation
            q_values = []
            for pert_action in perturbations:
                q_val = self.Q(state, pert_action)
                q_values.append(q_val.item())

            # Pick action with highest Q
            best_idx = np.argmax(q_values)
            action = perturbations[best_idx]

            return action

BCQ Core Mechanism

Batch Constraint Principle:

Without BCQ:
  Policy π can pick ANY action
  Problem: Picks actions not in dataset
  Result: Q overestimates for unseen actions

With BCQ:
  Policy π = β(s) + small_perturbation(s)
  Constraint: π must stay close to β
  Result: Only slightly improves on data actions

Why it works:
  - Q is accurate for β-like actions (in data)
  - Perturbations are small (confidence is high)
  - Policy can't escape to hallucinated region
  - Safe policy improvement

When BCQ is Appropriate

• Very limited data: Strict constraint needed
• Expert data with mistakes: Learn expert actions ± small perturbations
• Safety-critical: Can't risk exploring OOD actions
• Discrete action space: Easier to enumerate nearby actions

BCQ Pitfalls

• Too restrictive: Can't leverage good data effectively
• Assumes β is reasonable: If β is terrible, BCQ learns terrible policy
• Perturbation network complexity: Another network to train

Part 5: Distribution Shift and Offline Evaluation

Understanding Distribution Shift in Offline RL

The Core Challenge:

Training distribution:
  States visited by β (behavior policy)
  Actions β typically takes
  Transitions β experiences

Evaluation distribution:
  States visited by π (learned policy)
  Actions π learns to take
  Transitions π will experience

Problem:
  Training and evaluation distributions diverge
  π will visit states not in training data
  Q-estimates unreliable for those states

Offline Evaluation Without Environment Interaction

The Problem:

Online RL:
  π ← train(D)
  eval_return = run_policy(π, env, episodes=100)
  Simple and accurate

Offline RL:
  π ← train(D)
  Can't run in environment (offline constraint!)
  Must estimate return without interaction
  How to evaluate without cheating?

Offline Evaluation Methods

Method 1: Importance Sampling

def importance_sampling_eval(policy_pi, dataset_D):
    """
    Estimate E[G_π] using importance sampling.

    Key idea:
    E[G_π] = E_{(s,a,...)~π}[G]
           = E_{(s,a,...)~β}[G * (π(a|s)/β(a|s))]

    Use dataset transitions weighted by policy ratio.
    """
    total_return = 0
    total_weight = 0

    for trajectory in dataset_D:
        # Trajectory importance ratio
        traj_ratio = 1.0
        traj_return = 0

        for t, (s, a, r) in enumerate(trajectory):
            # Importance weight for this action
            pi_prob = policy_pi.log_prob(a, s).exp()
            beta_prob = behavior_policy.log_prob(a, s).exp()

            weight = pi_prob / (beta_prob + 1e-8)

            # Importance ratio for trajectory
            traj_ratio *= weight

            # Accumulate return
            traj_return += (0.99 ** t) * r

        # Weight trajectory by importance ratio
        total_return += traj_ratio * traj_return
        total_weight += traj_ratio

    # Average weighted return
    estimated_return = total_return / total_weight
    return estimated_return

Problem: High variance with small β probability. Weight explodes.

Method 2: Regression Importance Sampling (RIS)

def ris_eval(policy_pi, dataset_D, value_fn):
    """
    Regression Importance Sampling: combines IS with value function
    to reduce variance.

    Idea: Use learned V-function for long horizons, IS for short.
    """
    total_return = 0
    total_weight = 0

    for trajectory in dataset_D:
        for t, (s, a, r, s_next) in enumerate(trajectory):
            # Importance weight for this step
            pi_prob = policy_pi.log_prob(a, s).exp()
            beta_prob = behavior_policy.log_prob(a, s).exp()

            is_weight = pi_prob / (beta_prob + 1e-8)

            # Value of next state
            v_next = value_fn(s_next)

            # Hybrid estimate: IS-weighted reward + V-estimate of rest
            return_est = r + 0.99 * v_next
            weighted_return = is_weight * return_est

            total_return += weighted_return
            total_weight += is_weight

    return total_return / total_weight

Method 3: Model-based Estimation

def model_based_eval(policy_pi, dataset_D, dynamics_model):
    """
    Use learned dynamics model to estimate policy performance.

    Idea: π(s) → a, dynamics_model(s,a) → s', Q(s,a) → r
    """
    initial_states = dataset_D.sample_initial_states(batch_size=100)

    total_return = 0

    for s in initial_states:
        traj_return = 0
        done = False

        for t in range(horizon):
            # Policy action
            a = policy_pi.select_action(s)

            # Model prediction
            s_next = dynamics_model(s, a)
            r = reward_fn(s, a)  # or Q(s,a)

            # Accumulate return
            traj_return += (0.99 ** t) * r

            s = s_next
            if done:
                break

        total_return += traj_return

    estimated_return = total_return / len(initial_states)
    return estimated_return

Offline Evaluation Challenges

Challenge 1: Importance Weight Explosion
  - If β rarely takes action π prefers
  - IS weight = π(a|s)/β(a|s) becomes huge
  - Estimate has infinite variance

Challenge 2: V-Function Errors
  - If V-function incorrect
  - RIS estimates still wrong
  - Can't correct without environment feedback

Challenge 3: Model Errors
  - If dynamics model wrong
  - Model-based estimates diverge from reality
  - Especially bad for long horizons

Solution: Use multiple methods, cross-validate
  - If all methods agree: estimate is reliable
  - If methods disagree: be suspicious, try online validation

Part 6: When Offline RL is Needed

Decision Framework: Offline vs Online RL

Question 1: Can you collect more data?

YES → Consider online RL
  - Use environment interaction
  - Standard algorithms work well
  - Lower algorithmic complexity

NO → Offline RL necessary
  - Fixed dataset only
  - Learn without interaction
  - Handle overestimation explicitly

Question 2: Is data collection expensive?

YES (expensive) → Offline RL pays off
  - Robot experiments: $1000+ per hour
  - Medical trials: ethical constraints
  - Real-world deployment: safety concerns

NO (cheap) → Online RL usually better
  - Simulation available
  - Data generation easy
  - Self-play systems

Question 3: Is data collection dangerous?

YES → Offline RL + careful validation
  - Autonomous driving
  - Nuclear plants
  - Medical systems
  - Learn conservatively from past experience

NO → Online RL fine
  - Game environments
  - Safe simulators
  - Can afford exploration failures

When Offline RL is Essential

1. Real-World Robotics

Problem: Collect trajectory = robot experiment
Cost: $$$, time, expertise
Solution: Offline RL from logged demonstrations

Example: Learning from human tele-operation logs
- Data collection: humans control robot
- Training: offline RL from logs
- Deployment: robot improves on human behavior

Why offline RL helps:
- Can't try random actions (breaks hardware)
- Can't explore unsafely (danger)
- Limited budget for experiments

2. Medical Treatment Policies

Problem: Can't experiment on patients
Data: Historical patient records
Solution: Offline RL to find better treatments

Example: Learning antibiotic treatment policies
- Data: patient → treatment → outcome logs
- Training: offline RL from historical data
- Deployment: recommend treatments

Why offline RL helps:
- Can't do random exploration (unethical)
- Patient outcomes matter immediately
- Limited patient population

3. Recommendation Systems

Problem: Users leave if recommendations bad
Data: Historical user interactions
Solution: Offline RL to improve recommendations

Example: Movie recommendations
- Data: user watches movie → rates it
- Training: offline RL from interaction logs
- Deployment: recommend movies offline users will like

Why offline RL helps:
- Online experiment = worse user experience
- Can't A/B test extensively (business impact)
- Massive data available (can be offline)

When Online RL is Better

1. Simulation Available

Example: Atari games
- Infinite free samples
- Can explore safely
- Rewards deterministic
- Online RL solves it easily

Why offline RL unnecessary:
- Data collection cost ≈ 0
- Exploration safe (it's a game)
- Online algorithms highly optimized

2. Self-Play Systems

Example: Chess, Go
- Generate own data
- Unlimited exploration budget
- Learn strong policies easily
- Online RL natural

Why offline RL adds complexity:
- Data generation is free (self-play)
- Can afford to explore
- Online algorithms work better

3. Simulator Fidelity is High

Example: Training in simulation, deploy in reality
- Simulator accurate enough
- Can collect unlimited data
- Distribution shift minimal (sim matches reality)
- Online RL sufficient

Why offline RL unnecessary:
- Can collect all needed data in simulation
- Don't have distribution shift problem

Part 7: Common Pitfalls and Red Flags

Pitfall 1: Assuming Online Algorithm Will Work

Red Flag: "I'll just use DQN/PPO on my offline data."

Reality: Will overestimate values, learn poor policy.

Example:

Dataset: suboptimal human demonstrations
DQN trained on D: max Q estimated for unseen actions
Result: Policy picks actions never in dataset
Reality: actions fail in deployment

Correct approach:
- Use CQL/IQL to address overestimation
- Constrain policy to behavior support (BCQ)
- Evaluate carefully offline before deployment

Pitfall 2: Ignoring Distribution Shift

Red Flag: "Policy divergence shouldn't be a problem if data is diverse."

Reality: Even diverse data has gaps. Policy will find them.

Example:

Dataset: collected over 6 months, diverse actions
Your policy: learns to combine actions in novel ways
Result: visits unseen state combinations
Q-estimates fail for combinations not in data

Correct approach:
- Monitor policy divergence from data
- Use uncertainty estimates (ensemble Q-networks)
- Gradually deploy, validate offline metrics

Pitfall 3: Evaluating Offline with Online Metrics

Red Flag: "I trained the policy, let me just run it in the environment to evaluate."

Reality: That's not offline RL anymore! Defeats the purpose.

Example:

Offline RL goal: learn without environment interaction
Wrong evaluation: run π in environment 1000 times
Result: uses 1000s of samples for evaluation

Correct approach:
- Use offline evaluation methods (IS, RIS, model-based)
- Validate offline estimates before deployment
- Use conservative estimates (pessimistic evaluation)

Pitfall 4: Not Considering Data Quality

Red Flag: "My algorithm is robust, it handles any data."

Reality: Offline RL performance depends critically on data quality.

Example:

Good data: expert demonstrations, well-explored actions
CQL: performs well, conservative works fine

Bad data: random exploration, sparse rewards
CQL: learns very slowly (too pessimistic)
BCQ: learns random behavior (constrained to β)

Solution: Analyze data quality first
- Percent expert vs random actions
- Return distribution
- Action coverage
- Choose algorithm for data type

Pitfall 5: Overestimating Conservatism

Red Flag: "I'll use maximum pessimism to be safe."

Reality: Too much pessimism prevents learning anything.

Example:

Hyperparameter: CQL weight α = 1000 (extreme)
Result: Q(s,a) always negative (super pessimistic)
Policy: learns random actions (all have low Q)
Performance: no improvement over random

Correct approach:
- Tune conservatism to data quality
- Diverse data: less pessimism (CQL weight = 0.1)
- Limited data: more pessimism (CQL weight = 1.0)
- Validate offline evaluation metrics

Pitfall 6: Forgetting Batch Constraints

Red Flag: "CQL handles distribution shift, I don't need behavior cloning."

Reality: CQL addresses overestimation, not policy divergence.

Example:

CQL alone:
Q-values conservative (good)
Policy gradient: π(a|s) = exp(Q(s,a) / τ)
Policy can still diverge far from β (bad)
New states visited, Q unreliable

CQL + behavior cloning:
Q-values conservative
Policy constrained: π ≈ β + improvement
Policy stays near data
Safe exploration

Solution: Combine approaches
- CQL for value estimation
- KL constraint for policy divergence
- Behavior cloning for explicit constraint

Pitfall 7: Using Wrong Evaluation Metric

Red Flag: "I'll just report average Q-value as evaluation metric."

Reality: Q-values can be hallucinated!

Example:

High Q-values don't mean high returns
- If Q overestimates: high Q, low actual return
- If data is poor: high Q, bad actions

Correct metrics:
- Importance sampling estimate (IS)
- Regression IS estimate (RIS, lower variance)
- Model-based estimate (if model good)
- Conservative: min of multiple estimates

Pitfall 8: Not Handling Batch Imbalance

Red Flag: "I have 1M samples, that's enough data."

Reality: Sample diversity matters more than quantity.

Example:

Dataset composition:
- 900K samples: random actions
- 100K samples: expert demonstrations

Training:
- Q-network sees mostly random behavior
- Expert actions are rare
- Q overestimates expert actions (underfitting them)

Solution:
- Stratified sampling (balance expert/random)
- Importance weighting (weight rare samples higher)
- Separate Q-networks for different behavior types

Pitfall 9: Assuming Stationary Environment

Red Flag: "Environment hasn't changed, my offline policy should work."

Reality: Without online validation, you can't detect environment shifts.

Example:

Training period: robot arm dynamics stable
Deployment period: arm friction increased
Offline estimate: predicted 50 reward
Actual return: 20 reward

Why? Q-function trained on different dynamics.

Solution:
- Monitor deployment performance vs offline estimate
- If large gap: retrain on new data
- Use online validation after deployment

Pitfall 10: Not Accounting for Reward Uncertainty

Red Flag: "Rewards are fixed, no uncertainty."

Reality: Sparse/noisy rewards create uncertainty in Q-estimates.

Example:

Sparse reward environment:
- Most transitions have r = 0
- Few transitions have r = 1 (sparse signal)
- Q-function must extrapolate from few examples

CQL without reward uncertainty:
- Pessimistic on actions, not rewards
- Misses that some Q-high states might have low actual reward

Solution:
- Ensemble Q-networks (estimate uncertainty)
- Use epistemic uncertainty in policy improvement
- Conservative value averaging (use min over ensemble)

Part 8: Real-World Case Studies

Case Study 1: Robot Manipulation

Problem: Train robot to pick and place objects from demonstrations.

Data:
- 50 human teleoperation episodes (~500 transitions each)
- Expert actions: picking, placing, moving
- Cost: each episode = 1 hour robot time

Why offline RL:
- Data collection expensive (human time)
- Can't explore randomly (break hardware)
- Want to improve on expert (BC alone isn't enough)

Solution: IQL
- Learns from expert demonstrations
- V-function constraints policy to expert-like state distribution
- Iteratively improves on expert with confidence

Results:
- Offline training: 2 hours GPU
- Online validation: 1 hour robot time
- Policy: 50% success rate improvement over expert

Case Study 2: Recommendation Systems

Problem: Improve movie recommendations from historical user interactions.

Data:
- 100M user-movie interactions (1 year history)
- Reward: user watches movie (r=1) or skips (r=0)
- Distribution: mix of random recommendations and human editorial

Why offline RL:
- Online A/B test = worse user experience
- Can't explore randomly (hurts engagement metrics)
- But can learn better recommendations from behavior

Solution: CQL
- Conservatively estimates Q(user, movie)
- Avoids recommending movies only hypothetically good
- Safely recommends movies similar to successful past

Results:
- Offline metrics: 20% improvement over heuristics
- A/B test: 8% improvement (offline was too conservative)
- Trade-off: offline pessimism vs online optimality

Case Study 3: Medical Treatment Policy

Problem: Learn treatment policy from patient records to minimize mortality.

Data:
- 50K patient records (diverse treatments, outcomes)
- Actions: drug choices, dosages, procedures
- Reward: patient survives (r=1) or dies (r=-100)

Why offline RL:
- Can't experiment on patients (unethical)
- Sparse reward (most patients live)
- Distribution shift (changing patient population)

Solution: BCQ + IQL hybrid
- BCQ: only improve on treatments in data (avoid new untested drugs)
- IQL: expectile regression handles sparse rewards naturally
- Conservative deployment: start with small population, validate

Challenges:
- Data collection bias (sicker patients get more aggressive treatment)
- Measurement error (outcomes uncertain)
- Non-stationarity (medical practices evolve)

Results:
- Offline validatio

…(truncated)