Value Based Methods

Value-Based Methods

When to Use This Skill

Invoke this skill when you encounter:

• Algorithm Selection: "Should I use DQN or policy gradient for my problem?"
• DQN Implementation: User implementing DQN and needs guidance on architecture
• Training Issues: "DQN is diverging", "Q-values too high", "slow to learn"
• Variant Questions: "What's Double DQN?", "Should I use Dueling?", "Is Rainbow worth it?"
• Discrete Action RL: User has discrete action space and implementing value method
• Hyperparameter Tuning: Debugging learning rates, replay buffer size, network architecture
• Implementation Bugs: Target network missing, frame stacking wrong, reward scaling issues
• Custom Environments: Designing states, rewards, action spaces for DQN

This skill provides practical implementation guidance for discrete action RL.

Do NOT use this skill for:

• Continuous action spaces (route to actor-critic-methods)
• Policy gradients (route to policy-gradient-methods)
• Model-based RL (route to model-based-rl)
• Offline RL (route to offline-rl-methods)
• Theory foundations (route to rl-foundations)

Core Principle

Value-based methods solve discrete action RL by learning Q(s,a) = expected return from taking action a in state s, then acting greedily. They're powerful for discrete spaces but require careful implementation to avoid instability.

Key insight: Value methods assume you can enumerate and compare all action values. This breaks down with continuous actions (infinite actions to compare). Use them for:

• Games (Atari, Chess)
• Discrete control (robot navigation, discrete movement)
• Dialog systems (discrete utterances)
• Combinatorial optimization

Do not use for:

• Continuous control (robot arm angles, vehicle acceleration)
• Stochastic policies required (multi-agent, exploration in deterministic policy)
• Exploration of large action space (too slow to learn all actions)

Part 1: Q-Learning Foundation

From TD Learning to Q-Learning

You understand TD learning from rl-foundations. Q-learning extends it to action-values.

TD(0) for V(s):

V[s] ← V[s] + α(r + γV[s'] - V[s])

Q-Learning for Q(s,a):

Q[s,a] ← Q[s,a] + α(r + γ max_a' Q[s',a'] - Q[s,a])

Key difference: Q-learning has max over next actions (off-policy).

Off-Policy Learning

Q-learning learns the optimal policy π(a|s) = argmax_a Q(s,a)* regardless of exploration policy.

Example: Cliff Walking

Agent follows epsilon-greedy (explores 10% random)
But Q-learning learns: "Take safe path away from cliff" (optimal)
NOT: "Walk along cliff edge" (what exploring policy does sometimes)

Q-learning separates:
- Behavior policy: ε-greedy (for exploration)
- Target policy: greedy (what we're learning toward)

Why This Matters: Off-policy learning is sample-efficient (can learn from any exploration strategy). On-policy methods like SARSA would learn the exploration noise into policy.

Convergence Guarantee

Theorem: Q-learning converges to Q*(s,a) if:

1. All state-action pairs visited infinitely often
2. Learning rate α(t) → 0 (e.g., α = 1/N(s,a))
3. Sufficiently small ε (exploration not zero)

Practical: Use ε-decay schedule that ensures eventual convergence.

epsilon = max(epsilon_min, epsilon * decay_rate)
# Start: ε=1.0, decay to ε=0.01
# Ensures: all actions eventually tried, then exploitation takes over

Q-Learning Pitfall #1: Small State Spaces Only

Scenario: User implements tabular Q-learning for Atari.

Problem:

Atari image: 210×160 RGB = 20,160 pixels
Possible states: 256^20160 (astronomical)
Tabular Q-learning: impossible

Solution: Use function approximation (neural networks) → Deep Q-Networks

Red Flag: Tabular Q-learning works only for small state spaces (<10,000 unique states).

Part 2: Deep Q-Networks (DQN)

What DQN Adds to Q-Learning

DQN = Q-learning + neural network + two critical stability mechanisms:

1. Experience Replay: Break temporal correlation
2. Target Network: Prevent moving target problem

Mechanism 1: Experience Replay

Problem without replay:

# Naive approach (WRONG)
state = env.reset()
for t in range(1000):
    action = epsilon_greedy(state)
    next_state, reward = env.step(action)

    # Update Q from this single transition
    Q(state, action) += α(reward + γ max Q(next_state) - Q(state, action))
    state = next_state

Why this fails:

• Consecutive transitions are highly correlated (state_t and state_{t+1} very similar)
• Neural network gradient updates are unstable with correlated data
• Network overfits to recent trajectory

Experience Replay Solution:

# Collect experiences in buffer
replay_buffer = []

for episode in range(num_episodes):
    state = env.reset()
    for t in range(max_steps):
        action = epsilon_greedy(state)
        next_state, reward = env.step(action)

        # Store experience (not learn yet)
        replay_buffer.append((state, action, reward, next_state, done))

        # Sample random batch and learn
        if len(replay_buffer) > batch_size:
            batch = random.sample(replay_buffer, batch_size)
            for (s, a, r, s_next, done) in batch:
                if done:
                    target = r
                else:
                    target = r + gamma * max(Q(s_next))
                loss = (Q(s,a) - target)^2

            # Update network weights
            optimizer.step(loss)

        state = next_state

Why this works:

1. Breaks correlation: Random sampling decorrelates gradient updates
2. Sample efficiency: Reuse old experiences (learn more from same env interactions)
3. Stability: Averaged gradients are smoother

Mechanism 2: Target Network

Problem without target network:

# Moving target problem (WRONG)
loss = (Q(s,a) - [r + γ max Q(s_next, a_next)])^2
       #     ^^^^             ^^^^
       # Same network computing both target and prediction

Issue: Network updates move both the prediction AND the target, creating instability.

Analogy: Trying to hit a moving target that moves whenever you aim.

Target Network Solution:

# Separate networks
main_network = create_network()      # Learning network
target_network = create_network()    # Stable target (frozen)

# Training loop
loss = (main_network(s,a) - [r + γ max target_network(s_next)])^2
                                  ^^^^^^^^
                    Target network doesn't update every step

# Periodically synchronize
if t % update_frequency == 0:
    target_network = copy(main_network)  # Freeze for N steps

Why this works:

1. Stability: Target doesn't move as much (frozen for many steps)
2. Bellman consistency: Gives network time to learn, then adjusts target
3. Convergence: Bootstrapping no longer destabilized by moving target

DQN Architecture Pattern

import torch
import torch.nn as nn

class DQN(nn.Module):
    def __init__(self, input_size, num_actions):
        super().__init__()
        # For Atari: CNN backbone
        self.conv1 = nn.Conv2d(4, 32, kernel_size=8, stride=4)  # Frame stack: 4 frames
        self.conv2 = nn.Conv2d(32, 64, kernel_size=4, stride=2)
        self.conv3 = nn.Conv2d(64, 64, kernel_size=3, stride=1)

        # Flatten and FC layers
        self.fc1 = nn.Linear(64*7*7, 512)  # After convolutions
        self.fc_value = nn.Linear(512, 1)  # For dueling: value stream
        self.fc_actions = nn.Linear(512, num_actions)  # For dueling: advantage stream

    def forward(self, x):
        # x shape: (batch, 4, 84, 84) for Atari
        x = torch.relu(self.conv1(x))
        x = torch.relu(self.conv2(x))
        x = torch.relu(self.conv3(x))
        x = x.flatten(start_dim=1)
        x = torch.relu(self.fc1(x))

        # For basic DQN: just action values
        q_values = self.fc_actions(x)
        return q_values

Hyperparameter Guidance

DQN Pitfall #1: Missing Target Network

Symptom: "DQN loss explodes immediately, Q-values diverge to ±infinity"

Root cause: No target network (or target updates too frequently)

# WRONG - target network updates every step
loss = (Q(s,a) - [r + γ max Q(s_next)])^2  # Both from same network

# CORRECT - target network frozen for steps
loss = (Q_main(s,a) - [r + γ max Q_target(s_next)])^2
# Update target: if step % 1000 == 0: Q_target = copy(Q_main)

Fix: Verify target network update frequency (1000-5000 steps typical).

DQN Pitfall #2: Replay Buffer Too Small

Symptom: "Sample efficiency very poor, agent takes millions of steps to learn"

Root cause: Small replay buffer = replay many recent correlated experiences

# WRONG
replay_buffer_size = 10_000
# After 10k steps, only seeing recent experience (no diversity)

# CORRECT
replay_buffer_size = 100_000 or 1_000_000
# See diverse experiences from long history

Rule of Thumb: Replay buffer ≥ 10 × episode length (more is usually better)

Memory vs Sample Efficiency Tradeoff:

• 10k buffer: Low memory, high correlation (bad)
• 100k buffer: Moderate memory, good diversity (usually sufficient)
• 1M buffer: High memory, excellent diversity (overkill unless long episodes)

DQN Pitfall #3: No Frame Stacking

Symptom: "Learning very slow or doesn't converge"

Root cause: Single frame doesn't show velocity (violates Markov property)

# WRONG - single frame
state = current_frame  # No velocity information
# Network cannot infer: is ball moving left or right?

# CORRECT - stack frames
state = np.stack([frame_t, frame_{t-1}, frame_{t-2}, frame_{t-3}])
# Velocity: difference between consecutive frames

Implementation:

from collections import deque

class FrameBuffer:
    def __init__(self, num_frames=4):
        self.buffer = deque(maxlen=num_frames)

    def add_frame(self, frame):
        self.buffer.append(frame)

    def get_state(self):
        return np.stack(list(self.buffer))  # (4, 84, 84)

DQN Pitfall #4: Reward Clipping Wrong

Symptom: "Training unstable" or "Learned policy much worse than Q-values suggest"

Context: Atari papers clip rewards to {-1, 0, +1} for stability.

Misunderstanding: Clipping destroys reward information.

# WRONG - unthinking clip
reward = np.clip(reward, -1, 1)  # All rewards become -1,0,+1
# In custom env with rewards in [-100, 1000], loses critical information

# CORRECT - Normalize instead
reward = (reward - reward_mean) / reward_std
# Preserves differences, stabilizes scale

When to clip: Only if rewards are naturally in {-1, 0, +1} (like Atari).

When to normalize: Custom environments with arbitrary scales.

Part 3: Double DQN

The Overestimation Bias Problem

Max operator bias: In stochastic environments, max over noisy estimates is biased upward.

Example:

True Q*(s,a) values: [10.0, 5.0, 8.0]

Due to noise, estimates: [11.0, 4.0, 9.0]
                            ↑
                        True Q = 10, estimate = 11

Standard DQN takes max: max(Q_estimates) = 11
But true Q*(s,best_action) = 10

Systematic overestimation! Agent thinks actions better than they are.

Consequence:

• Inflated Q-values during training
• Learned policy (greedy) performs worse than Q-values suggest
• Especially bad early in training when estimates very noisy

Double DQN Solution

Insight: Use one network to select best action, another to evaluate it.

# Standard DQN (overestimates)
target = r + γ max_a Q_target(s_next, a)
         #        ^^^^
         # Both selecting and evaluating with same network

# Double DQN (unbiased)
best_action = argmax_a Q_main(s_next, a)      # Select with main network
target = r + γ Q_target(s_next, best_action)  # Evaluate with target network

Why it works:

• Decouples selection and evaluation
• Removes systematic bias
• Unbiased estimator of true Q*

Implementation

class DoubleDQN(DQNAgent):
    def compute_loss(self, batch):
        states, actions, rewards, next_states, dones = batch

        # Main network Q-values for current state
        q_values = self.main_network(states)
        q_values_current = q_values.gather(1, actions)

        # Double DQN: select action with main network
        next_q_main = self.main_network(next_states)
        best_actions = next_q_main.argmax(1, keepdim=True)

        # Evaluate with target network
        next_q_target = self.target_network(next_states)
        max_next_q = next_q_target.gather(1, best_actions).detach()

        # TD target (handles done flag)
        targets = rewards + (1 - dones) * self.gamma * max_next_q

        loss = F.smooth_l1_loss(q_values_current, targets)
        return loss

When to Use Double DQN

Use Double DQN if:

• Training a medium-complexity task (Atari)
• Suspicious that Q-values are too optimistic
• Want slightly better sample efficiency

Standard DQN is OK if:

• Small action space (less overestimation)
• Training is otherwise stable
• Sample efficiency not critical

Takeaway: Double DQN is strictly better, minimal cost, use it.

Part 4: Dueling DQN

Dueling Architecture: Separating Value and Advantage

Insight: Q(s,a) = V(s) + A(s,a) where:

• V(s): How good is this state? (independent of action)
• A(s,a): How much better is action a than average? (action-specific advantage)

Why separate:

1. Better feature learning: Network learns state features independently from action value
2. Stabilization: Value stream sees many states (more gradient signal)
3. Generalization: Advantage stream learns which actions matter

Example:

Atari Breakout:
V(s) = "Ball in good position, paddle ready" (state value)
A(s,LEFT) = -2 (moving left here hurts)
A(s,RIGHT) = +3 (moving right here helps)
A(s,NOOP) = 0 (staying still is neutral)

Q(s,LEFT) = V + A = 5 + (-2) = 3
Q(s,RIGHT) = V + A = 5 + 3 = 8  ← Best action
Q(s,NOOP) = V + A = 5 + 0 = 5

Architecture

class DuelingDQN(nn.Module):
    def __init__(self, input_size, num_actions):
        super().__init__()

        # Shared feature backbone
        self.conv1 = nn.Conv2d(4, 32, kernel_size=8, stride=4)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=4, stride=2)
        self.conv3 = nn.Conv2d(64, 64, kernel_size=3, stride=1)
        self.fc = nn.Linear(64*7*7, 512)

        # Value stream (single output)
        self.value_fc = nn.Linear(512, 256)
        self.value = nn.Linear(256, 1)

        # Advantage stream (num_actions outputs)
        self.advantage_fc = nn.Linear(512, 256)
        self.advantage = nn.Linear(256, num_actions)

    def forward(self, x):
        # Shared backbone
        x = torch.relu(self.conv1(x))
        x = torch.relu(self.conv2(x))
        x = torch.relu(self.conv3(x))
        x = x.flatten(start_dim=1)
        x = torch.relu(self.fc(x))

        # Value stream
        v = torch.relu(self.value_fc(x))
        v = self.value(v)

        # Advantage stream
        a = torch.relu(self.advantage_fc(x))
        a = self.advantage(a)

        # Combine: Q = V + (A - mean(A))
        # Subtract mean(A) for normalization (prevents instability)
        q = v + (a - a.mean(dim=1, keepdim=True))
        return q

Why Subtract Mean of Advantages?

# Without mean subtraction
q = v + a
# Problem: V and A not separately identifiable
# V could be 100 + A = -90 or V = 50 + A = -40 (same Q)

# With mean subtraction
q = v + (a - mean(a))
# Mean advantage = 0 on average
# Forces: V learns state value, A learns relative advantage
# More stable training

When to Use Dueling DQN

Use Dueling if:

• Training complex environments (Atari)
• Want better feature learning
• Training is unstable (helps stabilization)

Standard DQN is OK if:

• Simple environments
• Computational budget tight

Takeaway: Dueling is strictly better for neural network learning, minimal cost, use it.

Part 5: Prioritized Experience Replay

Problem with Uniform Sampling

Issue: All transitions equally likely to be sampled.

# Uniform sampling
batch = random.sample(replay_buffer, batch_size)
# Includes: boring transitions, important transitions, rare transitions
# All mixed together with equal weight

Problem:

• Wasted learning on transitions already understood
• Rare important transitions sampled rarely
• Sample inefficiency

Example:

Atari agent learns mostly: "Move paddle left-right in routine positions"
Rarely: "What happens when ball is in corner?" (rare, important)

Uniform replay: 95% learning about paddle, 5% about corners
Should be: More focus on corners (rarer, more surprising)

Prioritized Experience Replay Solution

Insight: Sample transitions proportional to TD error (surprise).

# Compute TD error (surprise)
td_error = |r + γ max Q(s_next) - Q(s,a)|
#           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#           How wrong was our prediction?

# Probability ∝ TD error^α
# High error transitions sampled more
batch = sample_proportional_to_priority(replay_buffer, priorities)

Implementation

import numpy as np

class PrioritizedReplayBuffer:
    def __init__(self, size, alpha=0.6):
        self.buffer = []
        self.priorities = []
        self.size = size
        self.alpha = alpha  # How much to prioritize (0=uniform, 1=full priority)
        self.epsilon = 1e-6  # Small value to avoid zero priority

    def add(self, experience):
        # New experiences get max priority (important!)
        max_priority = np.max(self.priorities) if self.priorities else 1.0

        if len(self.buffer) < self.size:
            self.buffer.append(experience)
            self.priorities.append(max_priority)
        else:
            # Replace oldest if full
            self.buffer[len(self.buffer) % self.size] = experience
            self.priorities[len(self.priorities) % self.size] = max_priority

    def sample(self, batch_size):
        # Compute sampling probabilities
        priorities = np.array(self.priorities) ** self.alpha
        priorities = priorities / np.sum(priorities)

        # Sample indices
        indices = np.random.choice(len(self.buffer), batch_size, p=priorities)
        batch = [self.buffer[i] for i in indices]

        # Importance sampling weights (correct for bias from prioritized sampling)
        weights = (1 / (len(self.buffer) * priorities[indices])) ** (1/3)  # β=1/3
        weights = weights / np.max(weights)  # Normalize

        return batch, indices, weights

    def update_priorities(self, indices, td_errors):
        # Update priorities based on new TD errors
        for idx, td_error in zip(indices, td_errors):
            self.priorities[idx] = (np.abs(td_error) + self.epsilon) ** self.alpha

Importance Sampling Weights

Problem: Prioritized sampling introduces bias (samples important transitions more).

Solution: Reweight gradients by inverse probability.

# Uniform sampling: each transition contributes equally
loss = mean((r + γ max Q(s_next) - Q(s,a))^2)

# Prioritized sampling: bias toward high TD error
# Correct with importance weight (large TD error → small weight)
loss = mean(weights * (r + γ max Q(s_next) - Q(s,a))^2)
#            ^^^^^^^
#      Importance sampling correction

# weights ∝ 1/priority (inverse)

When to Use Prioritized Replay

Use if:

• Training large environments (Atari)
• Sample efficiency critical
• Have computational budget for priority updates

Use standard uniform if:

• Small environments
• Computational budget tight
• Standard training is working fine

Note: Adds complexity (priority updates), minimal empirical gain in many cases.

Part 6: Rainbow DQN

Combining All Improvements

Rainbow = Double DQN + Dueling DQN + Prioritized Replay + 3 more innovations:

1. Double DQN: Reduce overestimation bias
2. Dueling DQN: Separate value and advantage
3. Prioritized Replay: Sample important transitions
4. Noisy Networks: Exploration through network parameters
5. Distributional RL: Learn Q distribution not just mean
6. Multi-step Returns: n-step TD learning instead of 1-step

When to Use Rainbow

Use Rainbow if:

• Need state-of-the-art Atari performance
• Have weeks of compute for tuning
• Paper requires it

Use Double + Dueling DQN if:

• Standard DQN training unstable
• Want good performance with less tuning
• Typical development

Use Basic DQN if:

• Learning the method
• Sample efficiency not critical
• Simple environments

Lesson: Understand components separately before combining.

Learning progression:
1. Q-learning (understand basics)
2. Basic DQN (add neural networks)
3. Double DQN (fix overestimation)
4. Dueling DQN (improve architecture)
5. Add prioritized replay (sample efficiency)
6. Rainbow (combine all)

Part 7: Common Bugs and Debugging

Bug #1: Training Divergence (Q-values explode)

Diagnosis Tree:

1. Check target network:

   # WRONG - updating every step
   loss = (Q_main(s,a) - [r + γ max Q_main(s_next)])^2
   # FIX - use separate target network
   loss = (Q_main(s,a) - [r + γ max Q_target(s_next)])^2
   

2. Check learning rate:

   # WRONG - too high
   optimizer = torch.optim.Adam(network.parameters(), lr=0.1)
   # FIX - reduce learning rate
   optimizer = torch.optim.Adam(network.parameters(), lr=0.0001)
   

3. Check reward scale:

   # WRONG - rewards too large
   reward = 1000 * indicator  # Values explode
   # FIX - normalize
   reward = 10 * indicator
   # Or: reward = (reward - reward_mean) / reward_std
   

4. Check replay buffer:

   # WRONG - too small
   replay_buffer_size = 1000
   # FIX - increase size
   replay_buffer_size = 100_000
   

Bug #2: Poor Sample Efficiency (Slow Learning)

Diagnosis Tree:

1. Check replay buffer size:

   # Too small → high correlation
   if len(replay_buffer) < 100_000:
       print("WARNING: Replay buffer too small for Atari")
   

2. Check target network update frequency:

   # Too frequent → moving target
   # Too infrequent → slow target adjustment
   # Good: every 1000-5000 steps
   if update_frequency > 10_000:
       print("Target updates too infrequent")
   

3. Check batch size:

   # Too small → noisy gradients
   # Too large → slow training
   # Good: 32-64
   if batch_size < 16 or batch_size > 256:
       print("Consider adjusting batch size")
   

4. Check epsilon decay:

   # Decaying too fast → premature exploitation
   # Decaying too slow → wastes steps exploring
   # Typical: decay over 10% of total steps
   if decay_steps < total_steps * 0.05:
       print("Epsilon decays too quickly")
   

Bug #3: Q-Values Too Optimistic (Learned Policy << Training Q)

Diagnosis:

Red Flag: Policy performance much worse than max Q-value during training.

# Symptom
max_q_value = 100.0
actual_episode_return = 5.0
# 20x gap suggests overestimation

# Solutions (try in order)
1. Use Double DQN (reduces overestimation)
2. Reduce learning rate (slower updates → less optimistic)
3. Increase target network update frequency (more stable target)
4. Check reward function (might be wrong)

Bug #4: Frame Stacking Wrong

Symptoms:

• Very slow learning despite "correct" implementation
• Network can't learn velocity-dependent behaviors

Diagnosis:

# WRONG - single frame
state_shape = (84, 84, 3)
# Network sees only position, not velocity

# CORRECT - stack 4 frames
state_shape = (84, 84, 4)
# Last 4 frames show motion

# Check frame stacking implementation
frame_stack = deque(maxlen=4)
for frame in frames:
    frame_stack.append(frame)
    state = np.stack(list(frame_stack))  # (4, 84, 84)

Bug #5: Network Architecture Mismatch

Symptoms:

• CNN on non-image input (or vice versa)
• Output layer wrong number of actions
• Input preprocessing wrong

Diagnosis:

# Image input → use CNN
if input_type == 'image':
    network = CNN(num_actions)

# Vector input → use FC
elif input_type == 'vector':
    network = FullyConnected(input_size, num_actions)

# Output layer MUST have num_actions outputs
assert network.output_size == num_actions

Part 8: Hyperparameter Tuning

Learning Rate

Too high (α > 0.001):

• Divergence, unstable training
• Q-values explode

Too low (α < 0.00001):

• Very slow learning
• May not converge in reasonable time

Start: α = 0.0001, adjust if needed

# Adaptive strategy
if max_q_value > 1000:
    print("Reduce learning rate")
    alpha = alpha / 2
if learning_curve_flat:
    print("Increase learning rate")
    alpha = alpha * 1.1

Replay Buffer Size

Too small (< 10k for Atari):

• High correlation in gradients
• Slow learning, poor sample efficiency

Too large (> 10M):

• Excessive memory
• Stale experiences dominate
• Diminishing returns

Rule of thumb: 10 × episode length

episode_length = 1000  # typical
ideal_buffer = 100_000  # 10 × typical Atari episode

# Can increase if GPU memory available and learning slow
if learning_slow:
    buffer_size = 500_000  # More diversity

Epsilon Decay

Too fast (decay in 10k steps):

• Agent exploits before learning
• Suboptimal policy

Too slow (decay in 1M steps):

• Wasted exploration time
• Slow performance improvement

Rule: Decay over ~10% of total training steps

total_steps = 1_000_000
epsilon_decay_steps = total_steps * 0.1  # 100k steps
epsilon = max(epsilon_min, epsilon * (epsilon_decay_steps / current_step))

Target Network Update Frequency

Too frequent (every 100 steps):

• Target still moves rapidly
• Less stabilization benefit

Too infrequent (every 100k steps):

• Network drifts far from target
• Large jumps in learning

Sweet spot: Every 1k-5k steps (1000 typical)

update_frequency = 1000  # steps between target updates
if update_frequency < 500:
    print("Target updates might be too frequent")
if update_frequency > 10_000:
    print("Target updates might be too infrequent")

Reward Scaling

No scaling (raw rewards vary wildly):

• Learning rate effects vary by task
• Convergence issues

Clipping (clip to {-1, 0, +1}):

• Good for Atari, loses information in custom envs

Normalization (zero-mean, unit variance):

• General solution
• Preserves reward differences

# Track running statistics
running_mean = 0.0
running_var = 1.0

def normalize_reward(reward):
    global running_mean, running_var
    running_mean = 0.99 * running_mean + 0.01 * reward
    running_var = 0.99 * running_var + 0.01 * (reward - running_mean)**2
    return (reward - running_mean) / np.sqrt(running_var + 1e-8)

Part 9: When to Use Each Method

DQN Selection Matrix

Action Space Check

Before implementing DQN, ask:

if action_space == 'continuous':
    print("ERROR: Use actor-critic or policy gradient")
    print("Value methods only for discrete actions")
    redirect_to_actor_critic_methods()

elif action_space == 'discrete' and len(actions) <= 100:
    print("✓ DQN appropriate")

elif action_space == 'discrete' and len(actions) > 1000:
    print("⚠ Large action space, consider policy gradient")
    print("Or: hierarchical RL, action abstraction")

Part 10: Red Flags Checklist

When you see these, suspect bugs:

• Single frame input: No velocity info, add frame stacking
• No target network: Divergence expected, add it
• Small replay buffer (< 10k): Poor efficiency, increase
• High learning rate (> 0.001): Instability likely, decrease
• No frame preprocessing: Raw image pixels, normalize to [0,1]
• Updating target every step: Moving target problem, freeze it
• No exploration decay: Explores forever, add epsilon decay
• Continuous actions: Wrong method, use actor-critic
• Very large rewards (> 100): Scaling issues, normalize
• Only one environment: Bias high, use frame skipping or multiple envs
• Immediate best performance: Overfitting to initial conditions, likely divergence later
• Q-values >> rewards: Overestimation, try Double DQN
• All Q-values zero: Network not learning, check learning rate
• Training loss increasing: Learning rate too high, divergence

Part 11: Pitfall Rationalization

Part 12: Pressure Test Scenarios

Scenario 1: Continuous Action Space

User: "I have a robot with continuous action space (joint angles in ℝ^7). Can I use DQN?"

Wrong Response: "Sure, discretize the actions" (Combinatorial explosion, inefficient)

Correct Response: "No, value methods are discrete-only. Use actor-critic (SAC) or policy gradient (PPO). They handle continuous actions naturally. Discretization would create 7-dimensional action space explosion (e.g., 10 values per joint = 10^7 actions)."

Scenario 2: Training Unstable

User: "My DQN is diverging immediately, loss explodes. Implementation looks right. What's wrong?"

Systematic Debug:

1. Check target network
   - Print: "Is target_network separate from main_network?"
   - Likely cause: updating together

2. Check learning rate
   - Print: "Learning rate = ?"
   - If > 0.001, reduce

3. Check reward scale
   - Print: "max(rewards) = ?"
   - If > 100, normalize

4. Check initial Q-values
   - Print: "mean(Q-values) = ?"
   - Should start near zero

Answer: Target network most likely culprit. Verify separate networks with proper update frequency.

Scenario 3: Rainbow vs Double DQN

User: "Should I implement Rainbow or just Double DQN? Is Rainbow worth the complexity?"

Guidance:

Double DQN:
+ Fixes overestimation bias
+ Simple to implement
+ 90% of Rainbow benefits in many cases
- Missing other optimizations

Rainbow:
+ Best Atari performance
+ State-of-the-art
- Complex (6 components)
- Harder to debug
- More hyperparameters

Recommendation:
Start: Double DQN
If unstable: Add Dueling
If slow: Add Prioritized
Only go to Rainbow: If need SotA and have time

Scenario 4: Frame Stacking Issue

User: "My agent trains on Atari but learning is slow. How many frames should I stack?"

Diagnosis:

# Check if frame stacking implemented
if state.shape != (4, 84, 84):
    print("ERROR: Not using frame stacking")
    print("Single frame (1, 84, 84) violates Markov property")
    print("Add frame stacking: stack last 4 frames")

# Frame count
4 frames: Standard (shows ~80ms at 50fps = ~4 frames)
3 frames: OK, slightly less velocity info
2 frames: Minimum, just barely Markovian
1 frame: WRONG, not Markovian
8+ frames: Too many, outdated states in stack

Scenario 5: Hyperparameter Tuning

User: "I've tuned learning rate, buffer size, epsilon. What else affects performance?"

Guidance:

Priority 1 (Critical):
- Target network update frequency (1000-5000 steps)
- Replay buffer size (100k+ typical)
- Frame stacking (4 frames)

Priority 2 (Important):
- Learning rate (0.0001-0.0005)
- Epsilon decay schedule (over ~10% of steps)
- Batch size (32-64)

Priority 3 (Nice to have):
- Network architecture (32-64-64 CNN standard)
- Reward normalization (helps but not required)
- Double/Dueling DQN (improvements, not essentials)

Start with Priority 1, only adjust Priority 2-3 if unstable.

Part 13: When to Route Elsewhere

Route to rl-foundations if

• User confused about Bellman equations
• Unclear on value function definition
• Needs theory behind Q-learning convergence

Route to actor-critic-methods if

• Continuous action space
• Need deterministic policy gradients
• Stochastic policy required

Route to policy-gradient-methods if

• Large discrete action space (> 1000 actions)
• Need policy regularization
• Exploration by stochasticity useful

Route to offline-rl-methods if

• No environment access (batch learning)
• Learning from logged data only

Route to rl-debugging if

• General training issues
• Need systematic debugging methodology
• Credit assignment problems

Route to reward-shaping if

• Sparse rewards
• Reward design affecting learning
• Potential-based shaping questions

Summary

You now understand:

1. Q-Learning: TD learning for action values, off-policy convergence guarantee
2. DQN: Add neural networks + experience replay + target network for stability
3. Stability Mechanisms:
   • Replay buffer: Break correlation
   • Target network: Prevent moving target problem
4. Common Variants:
   • Double DQN: Fix overestimation bias
   • Dueling DQN: Separate value and advantage
   • Prioritized Replay: Focus on important transitions
   • Rainbow: Combine improvements
5. When to Use: Discrete action spaces only, not continuous
6. Common Bugs: Divergence, poor efficiency, overoptimism, frame issues
7. Hyperparameter Tuning: Buffer size, learning rate, epsilon decay, target frequency
8. Debugging Strategy: Systematic diagnosis (target network → learning rate → reward scale)

Key Takeaways:

• Value methods are for discrete actions ONLY
• DQN requires target network and experience replay
• Frame stacking needed for video inputs (Markov property)
• Double DQN fixes overestimation, use it
• Start simple, add Dueling/Prioritized only if needed
• Systematic debugging beats random tuning

Next: Implement on simple environment first (CartPole or small custom task), then scale to Atari.