# Balanced Policy Optimization Rl > Stabilize off-policy RL for LLMs using adaptive clipping that dynamically rebalances positive/negative gradients and preserves entropy, improving mathematical reasoning performance vs standard PPO. - Skill: `adu2021/balanced-policy-optimization-rl` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/balanced-policy-optimization-rl` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/balanced-policy-optimization-rl/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/balanced-policy-optimization-rl --- # Technique: Balanced Policy Optimization — Adaptive Clipping for Stable LLM RL Off-policy reinforcement learning for LLMs faces two fundamental instability problems: positive-advantage samples get drowned out by negative-advantage samples during gradient updates, and fixed clipping (PPO-style) systematically suppresses entropy-increasing updates, leading to premature convergence and over-exploitation. BAPO solves both problems through **adaptive clipping**: instead of fixed clip ratios, dynamically adjust clipping bounds to balance gradients across positive and negative samples while explicitly preserving entropy. This enables more stable training and better final performance on complex reasoning tasks. ## Core Concept BAPO operates on three principles: - **Gradient Balancing**: Detect when negative samples dominate, adjust clipping to balance contributions - **Entropy Preservation**: Monitor when clipping suppresses entropy-increasing moves, override clipping for those updates - **Adaptive Thresholds**: Learn task-specific clipping ranges during training rather than using fixed values - **Theoretical Grounding**: Formalize as an "Entropy-Clip Rule" showing PPO's implicit entropy suppression The result is more stable training curves and stronger final performance on mathematical reasoning (+state-of-the-art on AIME). ## Architecture Overview - **Policy Network**: LLM backbone producing action probabilities - **Advantage Estimator**: Compute advantage from rewards or returns - **Adaptive Clipper**: Monitor gradient distributions, adjust clip bounds dynamically - **Entropy Monitor**: Track KL divergence, detect when clipping suppresses exploration - **Gradient Aggregator**: Combine clipped and unclipped gradients intelligently - **Training Loop**: Standard RL trajectory collection + BAPO-modified policy updates ## Implementation Steps The key innovation is replacing fixed PPO clipping with dynamic bounds that balance gradients. This example shows the core algorithm. ```python import torch import torch.nn as nn from typing import Tuple class BalancedPolicyOptimizer: """ Adaptive clipping for stable off-policy RL training. """ def __init__( self, model: nn.Module, initial_clip_ratio: float = 0.2, entropy_weight: float = 0.01, gradient_balance_target: float = 1.0 ): self.model = model self.clip_ratio = initial_clip_ratio self.entropy_weight = entropy_weight self.gradient_balance_target = gradient_balance_target self.optimizer = torch.optim.Adam(model.parameters(), lr=1e-5) def compute_advantages( self, rewards: torch.Tensor, values: torch.Tensor, gamma: float = 0.99, gae_lambda: float = 0.95 ) -> torch.Tensor: """ Compute generalized advantage estimates (GAE). Args: rewards: (batch, seq_len) values: (batch, seq_len) value estimates Returns: advantages: (batch, seq_len) """ advantages = torch.zeros_like(rewards) gae = 0.0 for t in reversed(range(len(rewards))): if t == len(rewards) - 1: next_value = 0.0 else: next_value = values[:, t + 1] delta = rewards[:, t] + gamma * next_value - values[:, t] gae = delta + gamma * gae_lambda * gae advantages[:, t] = gae return advantages def compute_policy_loss_with_adaptive_clipping( self, old_log_probs: torch.Tensor, new_log_probs: torch.Tensor, advantages: torch.Tensor, entropy: torch.Tensor ) -> Tuple[torch.Tensor, dict]: """ Compute policy loss with adaptive clipping. """ # Compute ratio: importance weight for off-policy correction ratio = torch.exp(new_log_probs - old_log_probs) # Separate positive and negative advantages positive_mask = advantages > 0 negative_mask = advantages <= 0 # Standard PPO clipping clipped_ratio = torch.clamp(ratio, 1 - self.clip_ratio, 1 + self.clip_ratio) # Compute losses for positive and negative samples separately positive_loss = torch.where( positive_mask, torch.min(ratio * advantages, clipped_ratio * advantages), torch.zeros_like(advantages) ) negative_loss = torch.where( negative_mask, torch.max(ratio * advantages, clipped_ratio * advantages), torch.zeros_like(advantages) ) # Check gradient dominance: are negatives suppressing positives? positive_grad = positive_loss.sum() negative_grad = -negative_loss.sum() # Negative because they decrease loss gradient_ratio = torch.abs(negative_grad) / (torch.abs(positive_grad) + 1e-8) # Adaptive clipping: if negatives dominate, loosen clipping for them if gradient_ratio > self.gradient_balance_target: # Increase clip range for negatives to let them contribute less adaptive_clip_negative = self.clip_ratio * (1.0 + 0.5 * gradient_ratio) adaptive_ratio_neg = torch.clamp( ratio, 1 - adaptive_clip_negative, 1 + adaptive_clip_negative ) negative_loss = torch.where( negative_mask, torch.max(ratio * advantages, adaptive_ratio_neg * advantages), torch.zeros_like(advantages) ) # Combine losses policy_loss = -(positive_loss + negative_loss).mean() # Entropy bonus: encourage exploration entropy_loss = -self.entropy_weight * entropy.mean() total_loss = policy_loss + entropy_loss return total_loss, { "policy_loss": policy_loss.item(), "entropy_bonus": entropy_loss.item(), "gradient_ratio": gradient_ratio.item(), "adaptive_clip": self.clip_ratio } def update_adaptive_clip_ratio(self, gradient_ratio: float): """ Adjust clipping ratio based on gradient balance. Higher gradient_ratio -> more negative dominance -> increase clip range. """ if gradient_ratio > self.gradient_balance_target: # Negative samples dominate: increase clip ratio to let positives breathe self.clip_ratio = min(self.clip_ratio * 1.1, 0.5) elif gradient_ratio < self.gradient_balance_target * 0.5: # Positive samples dominate: decrease clip ratio to reduce variance self.clip_ratio = max(self.clip_ratio * 0.9, 0.1) def train_step_with_bapo( model: nn.Module, optimizer: BalancedPolicyOptimizer, old_log_probs: torch.Tensor, new_log_probs: torch.Tensor, rewards: torch.Tensor, values: torch.Tensor, entropy: torch.Tensor ) -> dict: """ Single training step with BAPO. """ # Compute advantages advantages = optimizer.compute_advantages(rewards, values) # Normalize advantages for stability advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8) # Compute loss with adaptive clipping loss, metrics = optimizer.compute_policy_loss_with_adaptive_clipping( old_log_probs, new_log_probs, advantages, entropy ) # Update model optimizer.optimizer.zero_grad() loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) optimizer.optimizer.step() # Adapt clipping ratio for next step optimizer.update_adaptive_clip_ratio(metrics["gradient_ratio"]) return metrics ``` The key innovation is monitoring gradient contributions from positive vs negative samples and dynamically adjusting clipping to balance them. This prevents negative samples from suppressing beneficial positive updates. ## Practical Guidance | Setting | Metric | BAPO | Standard PPO | |---------|--------|------|-------------| | AIME 2024 | Accuracy | 59.6% | 52.1% | | AIME 2025 | Accuracy | 55.4% | 48.3% | | Training stability | Variance | Lower | Higher | **When to Use:** - Off-policy RL training for LLMs (reasoning, coding) - Training instability or premature convergence observed - Mathematical reasoning or complex problem-solving tasks - You need stronger final performance than PPO achieves **When NOT to Use:** - Online RL (on-policy methods preferred for stability) - Simple tasks where PPO works well - Extreme compute constraints (BAPO adds negligible cost) - Non-reasoning tasks (advantages may not separate clearly) **Common Pitfalls:** - Gradient balance target too strict → model converges prematurely - Not normalizing advantages → unstable gradient signal - Entropy weight too low → insufficient exploration - Clip ratio bounds too loose → uncontrolled variance - Not monitoring gradient ratios during training → miss convergence issues ## Reference [BAPO: Stabilizing Off-Policy RL for LLMs via Balanced Policy Optimization with Adaptive Clipping](https://arxiv.org/abs/2510.18927)