Pufferlib V3

PufferLib v3

Use this skill for PufferLib 3.x (pufferlib>=3.0,<4.0). PufferLib 3.0 was released on PyPI on June 23, 2025. The public v3 docs emphasize the puffer CLI, pufferlib.pufferl, Ocean environments, configurable vectorization, and PPO/PufferRL training.

[!IMPORTANT] Do not use the deprecated v1 top-level import from pufferlib import PuffeRL. In v3 examples, import the training module with from pufferlib import pufferl, then construct pufferl.PuffeRL(...) or use the puffer train/eval/sweep CLI.

Version Selection

• Prefer pufferlib-v3 for pufferlib>=3.0,<4.0, the June 2025 PyPI release line, and docs that show pufferl.load_config, pufferl.load_env, or pufferl.load_policy.
• Prefer pufferlib-v2 only for code pinned to pufferlib>=2.0,<3.0 or reproducing the 1M SPS Ocean release.
• Prefer the legacy pufferlib skill only for pufferlib==1.0.0.
• If a project tracks the 2026 4.0 branch or puffer.ai/docs.html, inspect upstream first; do not assume v3 signatures.

Installation

python -m venv .venv
source .venv/bin/activate
pip install "pufferlib>=3.0,<4.0"

For CUDA/native builds, use the project's lockfile or upstream PufferTank setup. PufferLib uses C/CUDA extensions for fast training paths, so mismatched PyTorch/CUDA/build isolation can produce import or build errors.

Fast CLI Workflow

Use the CLI first when working with built-in environments:

puffer train ENV_NAME --help
puffer train puffer_breakout
puffer eval puffer_breakout --help
puffer sweep puffer_breakout --help

Common option namespaces are train, env, vec, policy, and sweep-related groups. Prefer --help in the active environment because exact options can vary by installed version and environment config.

Python Training Workflow

Use this v3 shape for programmatic training:

import pufferlib.ocean
import pufferlib.vector
from pufferlib import pufferl


def train(env_name='puffer_breakout'):
    args = pufferl.load_config(env_name)
    args['train']['total_timesteps'] = 10_000_000
    args['train']['learning_rate'] = 0.001
    args['vec']['num_workers'] = 4

    vecenv = pufferl.load_env(env_name, args)
    policy = pufferl.load_policy(args, vecenv, env_name)
    trainer = pufferl.PuffeRL(args['train'], vecenv, policy)

    while trainer.epoch < trainer.total_epochs:
        trainer.evaluate()
        logs = trainer.train()
        if logs:
            print(logs)

    trainer.print_dashboard()
    trainer.close()


if __name__ == '__main__':
    train()

Adjust the config dictionary before constructing the environment/policy/trainer. Keep environment-specific defaults from load_config unless you know why to override them.

Custom Policy Pattern

Policies are PyTorch modules. For non-recurrent discrete-action policies, implement forward(observations, state=None) and return (logits, values):

import torch
import pufferlib.pytorch


class Policy(torch.nn.Module):
    def __init__(self, env):
        super().__init__()
        obs_size = env.single_observation_space.shape[0]
        action_size = env.single_action_space.n
        self.encoder = torch.nn.Sequential(
            pufferlib.pytorch.layer_init(torch.nn.Linear(obs_size, 128)),
            torch.nn.ReLU(),
            pufferlib.pytorch.layer_init(torch.nn.Linear(128, 128)),
            torch.nn.ReLU(),
        )
        self.actor = pufferlib.pytorch.layer_init(torch.nn.Linear(128, action_size), std=0.01)
        self.critic = pufferlib.pytorch.layer_init(torch.nn.Linear(128, 1), std=1.0)

    def forward(self, observations, state=None):
        hidden = self.encoder(observations)
        return self.actor(hidden), self.critic(hidden)

    def forward_eval(self, observations, state=None):
        return self.forward(observations, state)

For recurrent policies, preserve the state argument and follow repository examples for LSTM hidden-state keys. Do not silently drop masks, done flags, or environment IDs if the trainer passes them.

Vectorization Workflow

import gymnasium
import pufferlib.emulation
import pufferlib.vector


def make_env():
    env = gymnasium.make('CartPole-v1')
    return pufferlib.emulation.GymnasiumPufferEnv(env)

vecenv = pufferlib.vector.make(
    make_env,
    num_envs=8,
    num_workers=4,
    batch_size=2,
    backend=pufferlib.vector.Multiprocessing,
)

Guidance:

• Use pufferlib.vector.Serial for debugging and pufferlib.vector.Multiprocessing for throughput.
• Keep num_envs, num_workers, and batch_size compatible with the installed version's divisibility rules.
• Prefer native Ocean environments for maximum throughput; use Gymnasium/PettingZoo wrappers for compatibility and migration.
• Validate reset/step shapes in serial mode before multiprocessing.

Distributed Training

For multi-GPU v3 runs, use PyTorch distributed launch patterns only after a single-process run works:

torchrun --standalone --nnodes=1 --nproc-per-node=NUM_GPUS -m pufferlib.pufferl train ENV_NAME

Check that checkpoints/log directories are rank-safe and that only rank 0 writes user-facing artifacts unless upstream examples do otherwise.

Migration from v1/v2

• Replace from pufferlib import PuffeRL with from pufferlib import pufferl.
• Replace hand-rolled config objects with pufferl.load_config(env_name) where possible.
• Replace old environment factory code with pufferl.load_env(...) for built-ins, or pufferlib.vector.make(...) for custom factories.
• Ensure policies accept state=None and return logits plus values.
• Re-run CLI --help after upgrades; config namespaces and option names are version-sensitive.

Debugging Checklist

• Trainer import fails: verify package version and import from pufferlib import pufferl.
• C/CUDA extension import fails: check PyTorch/CUDA compatibility and try the upstream container or install instructions for non-default PyTorch.
• Multiprocessing hangs: reproduce with Serial, then one worker, then scale up.
• Action space errors: inspect vecenv.single_action_space; continuous and structured action support can require policy/output changes.
• Poor learning despite high SPS: sweep learning rate first, then rollout horizon/batch size/minibatch size, entropy coefficient, gamma, and lambda.

Sources to Check When Updating

• PyPI release history for exact 3.x release dates.
• PufferLib v3 Mintlify docs for CLI, quickstart, models, vectorization, and distributed training.
• Current upstream puffer.ai/docs.html if the project is not pinned; that site may describe newer 4.x behavior.