Paper Rob Resfit

ResFiT: Residual Off-Policy RL for Finetuning Behavior Cloning Policies

Ankile, Jiang, Duan, Shi, Abbeel, Nagabandi (Stanford / Amazon FAR / CMU / UC Berkeley, 2025)

What It Does

ResFiT finetunes pre-trained BC policies by learning lightweight per-step residual corrections via off-policy RL (TD3-based). The BC policy is treated as a frozen black box; a residual actor outputs small additive corrections (action_scale * tanh(...)) that are summed with BC actions before environment execution.

Key results:

• 200x more sample-efficient than on-policy (PPO) approaches (200k vs 40M steps)
• First real-world RL training on a humanoid robot with dexterous hands (29-DoF Vega)
• BoxClean: 98% success vs 77% BC baseline

Method Overview

                    obs
                     |
            +--------+--------+
            |                 |
     [BC Policy (frozen)]  [RL Residual Actor]
            |                 |
        base_action     residual_action (scaled by action_scale, e.g. 0.1-0.2)
            |                 |
            +--------+--------+
                     |
            combined = clamp(base + residual, -1, 1)
                     |
                [Environment]
                     |
              sparse reward {0, 1}

Two-phase training:

1. Critic warmup (critic_warmup_steps): Only train critic, actor frozen. Random exploration noise around base policy.
2. Joint training: Train both critic and actor with off-policy TD3 + n-step returns.

Key design choices:

• UTD ratio = 4 (updates per env step)
• N-step returns (n=5) for sparse reward propagation
• 10-head critic ensemble (RED-Q style), min over 2 random heads for target
• Layer normalization throughout
• Offline demo mixing (50% offline / 50% online by default, RLPD-style)

Architecture

Paper-Code Mapping

Key Classes & Signatures

BasePolicyVecEnvWrapper

# resfit/rl_finetuning/wrappers/residual_env_wrapper.py
class BasePolicyVecEnvWrapper:
    def __init__(self, vec_env, base_policy: ACTPolicy, action_scaler, state_standardizer):
        ...
    def reset(self) -> tuple[dict[str, Tensor], dict]:
        # Returns augmented obs with "observation.base_action" key
    def step(self, residual_naction: Tensor) -> tuple[dict, Tensor, Tensor, Tensor, dict]:
        # combined = base_naction + residual_naction, then unscale for env

QAgent

# resfit/rl_finetuning/off_policy/rl/q_agent.py
class QAgent(nn.Module):
    def __init__(self, obs_shape, prop_shape, action_dim, rl_cameras, cfg: QAgentConfig, residual_actor=True):
        ...
    def act(self, obs: dict[str, Tensor], eval_mode=False, stddev=0.0, cpu=True) -> Tensor:
        # Returns residual action
    def update(self, batch, stddev, update_actor, bc_batch=None, ref_agent=None) -> dict:
        # Full critic + actor update step

Actor (Residual Mode)

# resfit/rl_finetuning/off_policy/rl/actor.py
class Actor(nn.Module):
    def __init__(self, repr_dim, patch_repr_dim, prop_dim, action_dim, cfg: ActorConfig, residual_actor=True):
        # When residual_actor=True, takes base_action as additional input
    def forward(self, obs: dict, std: float) -> TruncatedNormal:
        # Returns distribution over scaled_mu = tanh(policy(x)) * action_scale

Config Hierarchy

ResidualTD3DexmgConfig          # Top-level experiment config
├── algo: ResidualTD3AlgoConfig  # Training hyperparams
│   ├── total_timesteps: 300_000
│   ├── batch_size: 256
│   ├── n_step: 3 (override to 5)
│   ├── gamma: 0.99 (override to 0.995)
│   ├── num_updates_per_iteration: 4  (UTD ratio)
│   ├── critic_warmup_steps: 10_000
│   ├── learning_starts: 10_000
│   ├── offline_fraction: 0.5
│   ├── stddev_max/min: 0.05
│   └── random_action_noise_scale: 0.2
├── agent: QAgentConfig
│   ├── actor_lr: 1e-6
│   ├── critic_lr: 1e-4
│   ├── critic_target_tau: 0.005
│   └── actor: ActorConfig
│       ├── action_scale: 0.1 (override to 0.2)
│       └── actor_last_layer_init_scale: 0.0  # Critical: start residual at zero
├── base_policy: BasePolicyConfig
│   └── wandb_id: "<project>/<run_id>"
├── offline_data: OfflineDataConfig
│   └── name: "ankile/dexmg-two-arm-coffee" (HuggingFace dataset)
└── wandb: WandBConfig

Key Hyperparameters

Dependencies

python==3.10
torch (CUDA 12.8)
torchrl==0.9.2
draccus==0.10.0
hydra-core
omegaconf
lerobot  # HuggingFace LeRobot
wandb
mujoco (via robosuite/DexMG environments)

Installation

conda create -n residual python=3.10 -y && conda activate residual
git clone https://github.com/amazon-far/residual-offpolicy-rl
cd residual-offpolicy-rl

# Core deps
./resfit/rl_finetuning/setup_rlpd_robosuite.sh
pip install wandb draccus==0.10.0 torchrl==0.9.2 hydra-core serial deepdiff matplotlib

# Auth
huggingface-cli login
wandb login

Gotchas & Tips

• actor_last_layer_init_scale=0.0 is critical -- ensures the residual starts at zero so the combined policy equals the BC policy at init
• The environment wrapper (BasePolicyVecEnvWrapper) handles the base+residual action combination -- the training loop only sees residual actions
• Images stored as uint8 in replay buffer to save GPU memory; converted to float on the fly
• Only num_envs=1 is supported due to n-step return implementation
• Base policy weights loaded from W&B artifacts -- you need to train BC first and log to W&B
• The action_scaler normalizes actions to [-1, 1] range for both base and residual policies