Reinforcement Learning

Reinforcement Learning Best Practices

Overview

This skill provides comprehensive guidance for implementing reinforcement learning in Python using the modern ecosystem (Gymnasium >= 1.0, Stable-Baselines3 >= 2.x). Gymnasium has replaced OpenAI Gym as the standard environment interface. Stable-Baselines3 (SB3) is recommended for prototyping, RLlib for production/distributed training, and CleanRL for research.

When to Use

• Building RL agents for discrete or continuous control tasks
• Creating custom simulation environments
• Tuning hyperparameters for RL algorithms
• Debugging training issues (reward curves, policy collapse, numerical instability)
• Deploying trained policies to production

Library Selection

Algorithm Decision Tree

Start
  |
  v
Action space type?
  |
  +-- Discrete --> Sample efficiency critical?
  |                  |
  |                  +-- Yes --> DQN (or Double/Dueling DQN)
  |                  +-- No  --> Stability critical?
  |                               |
  |                               +-- Yes --> PPO
  |                               +-- No  --> A2C (faster iterations)
  |
  +-- Continuous --> Sample efficiency critical?
                       |
                       +-- Yes --> SAC (auto entropy) or TD3
                       +-- No  --> PPO (more stable, less efficient)

Quick Selection Table:

Quick Start with Stable-Baselines3

Basic Training

from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env

# Create vectorized environment (4 parallel envs)
env = make_vec_env("CartPole-v1", n_envs=4)

# Initialize and train
model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=100_000)

# Save and load
model.save("ppo_cartpole")
loaded_model = PPO.load("ppo_cartpole")

# Evaluate
obs = env.reset()
for _ in range(1000):
    action, _ = loaded_model.predict(obs, deterministic=True)
    obs, reward, done, info = env.step(action)

Custom Environment Template

import gymnasium as gym
from gymnasium import spaces
import numpy as np

class CustomEnv(gym.Env):
    metadata = {"render_modes": ["human", "rgb_array"]}

    def __init__(self, render_mode=None):
        super().__init__()
        self.observation_space = spaces.Box(
            low=-np.inf, high=np.inf, shape=(4,), dtype=np.float32
        )
        self.action_space = spaces.Discrete(2)
        self.render_mode = render_mode

    def reset(self, seed=None, options=None):
        super().reset(seed=seed)
        self.state = self.np_random.uniform(low=-0.05, high=0.05, size=(4,))
        return self.state.astype(np.float32), {}

    def step(self, action):
        # Implement environment dynamics here
        observation = self.state.astype(np.float32)
        reward = 1.0
        terminated = False  # Episode ended due to task completion/failure
        truncated = False   # Episode ended due to time limit
        info = {}
        return observation, reward, terminated, truncated, info

    def render(self):
        pass

Hyperparameter Tuning with Optuna

import optuna
from stable_baselines3 import PPO
from stable_baselines3.common.evaluation import evaluate_policy

def objective(trial):
    learning_rate = trial.suggest_float("learning_rate", 1e-5, 1e-3, log=True)
    n_steps = trial.suggest_categorical("n_steps", [256, 512, 1024, 2048])
    gamma = trial.suggest_float("gamma", 0.9, 0.9999)

    model = PPO(
        "MlpPolicy", "CartPole-v1",
        learning_rate=learning_rate,
        n_steps=n_steps,
        gamma=gamma,
        verbose=0
    )
    model.learn(total_timesteps=50_000)

    mean_reward, _ = evaluate_policy(model, model.get_env(), n_eval_episodes=10)
    return mean_reward

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=50)
print(f"Best params: {study.best_params}")

Core Workflow

1. Define the environment - Use Gymnasium API
2. Validate the environment - Run python scripts/validate_env.py <env> (API compliance, spaces, determinism, NaN guards)
3. Select algorithm - Based on action space and requirements
4. Pre-training sanity check - Run python scripts/pretrain_check.py <env> (random-policy baseline, distribution stats, actionable warnings)
5. Start simple - Default hyperparameters, short training
6. Monitor training - TensorBoard, check reward curves
7. Debug issues - Use the debugging playbook
8. Tune hyperparameters - Optuna for systematic search
9. Evaluate properly - Run python scripts/eval_report.py <env> --model <model.zip> (separate eval env, per-episode success tracking, Markdown report)
10. Deploy - Export to ONNX/TorchScript

Bundled Scripts (run these instead of re-deriving snippets)

The validation and evaluation patterns from the reference files are packaged as runnable, self-contained CLIs in scripts/. Run them directly rather than re-implementing the inline snippets:

All three accept either a registered Gymnasium id (CartPole-v1) or a module.path:EnvClass spec for unregistered custom environments. validate_env.py exits nonzero on any failed check, so it works as a CI gate. eval_report.py without --model evaluates a random policy - useful as a baseline and for testing the report pipeline.

Reference Files

• algorithms.md - Deep dive on DQN, PPO, SAC, A2C, TD3
• environments.md - Gymnasium setup, custom envs, wrappers
• training.md - Hyperparameters, reward engineering, normalization
• debugging.md - Failure modes, diagnostics, sanity checks
• evaluation.md - Metrics, logging, reproducibility
• deployment.md - ONNX export, inference optimization, safety

Essential Dependencies

pip install gymnasium stable-baselines3 tensorboard optuna
# For Atari environments
pip install gymnasium[atari] gymnasium[accept-rom-license]
# For MuJoCo
pip install gymnasium[mujoco]

Common Pitfalls to Avoid

1. Not normalizing observations - Use VecNormalize wrapper
2. Wrong action space handling - Check discrete vs continuous
3. Ignoring seed management - Set seeds for reproducibility
4. Training and eval on same env - Use separate eval environment
5. Not monitoring entropy - Low entropy = policy collapse
6. Sparse rewards without shaping - Add intermediate rewards
7. Too large/small learning rate - Start with 3e-4 for most algorithms