# High Entropy Minority Tokens Rl > Optimize only high-entropy tokens during RL training to achieve better reasoning performance with 80% fewer gradient updates. - Skill: `adu2021/high-entropy-minority-tokens-rl` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/high-entropy-minority-tokens-rl` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/high-entropy-minority-tokens-rl/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/high-entropy-minority-tokens-rl --- # Focus RL Training on Decision Points, Not Repetition During chain-of-thought reasoning, most tokens are predictable continuations of established patterns. A small fraction—high-entropy tokens at logical branching points—actually determine which reasoning path the model takes. Standard RL training updates all tokens equally, wasting compute on tokens that barely vary across samples. This skill teaches selective gradient updates: identify and optimize only the high-entropy tokens that drive diverse reasoning outcomes, achieving superior performance with 5x fewer parameter updates. The insight is that token entropy reveals decision points. When entropy is high, the model genuinely considers multiple paths; when low, the token is predetermined. By concentrating learning signals on these critical decision points, you maximize the impact of each gradient update and prevent the model from overfitting to low-entropy repetitive patterns. ## Core Concept In typical token generation, entropy varies dramatically: deterministic tokens (articles, common words) have near-zero entropy, while decision-bearing tokens (logical operators, structure choices) have high entropy. Standard reinforcement learning treats all tokens equally, computing gradients for both the predictable and the consequential. Selective gradient updates invert this: compute gradients only for the minority of high-entropy tokens that actually influence which reasoning path is taken. This concentrates learning signal and reduces gradient noise from tokens that aren't truly "choosing" anything. ## Architecture Overview - **Entropy Analysis Module**: Computes token-level entropy across rollouts to identify decision points - **Token Filtering**: Masks low-entropy tokens and focuses optimization on high-entropy subset (typically 15-25% of tokens) - **Selective PPO/REINFORCE**: Standard RL algorithm but with gradients computed only for high-entropy positions - **Adaptive Threshold**: Entropy threshold adjusts based on task; math reasoning needs higher threshold than code - **Verification Integration**: Works with verifiable reward signals from solvers or evaluators ## Implementation This implementation demonstrates entropy-based token filtering and selective gradient updates for LLM RL training. First, analyze entropy patterns to identify high-entropy tokens: ```python import torch import torch.nn.functional as F import numpy as np from typing import List, Tuple from transformers import AutoTokenizer, AutoModelForCausalLM class TokenEntropyAnalyzer: """Identify high-entropy decision tokens in LLM outputs.""" def __init__(self, model_name: str = "gpt2-medium"): self.tokenizer = AutoTokenizer.from_pretrained(model_name) self.model = AutoModelForCausalLM.from_pretrained(model_name) def compute_token_entropy(self, prompt: str, generated_ids: List[int]) -> np.ndarray: """ Compute entropy for each token in a generation. High entropy = LLM was uncertain; low entropy = LLM was confident. """ input_ids = self.tokenizer.encode(prompt, return_tensors="pt") with torch.no_grad(): outputs = self.model(input_ids, output_hidden_states=True) logits = outputs.logits[0] # [seq_len, vocab_size] # Compute entropy for each position probs = F.softmax(logits, dim=-1) entropy = -(probs * torch.log(probs + 1e-10)).sum(dim=-1) return entropy.cpu().numpy() def identify_high_entropy_tokens(self, prompt: str, generated_ids: List[int], percentile: float = 75.0) -> np.ndarray: """ Identify tokens in top percentile of entropy (decision points). Returns binary mask: 1 = high entropy (optimize), 0 = low entropy (skip). """ entropy = self.compute_token_entropy(prompt, generated_ids) # Threshold: tokens above percentile threshold = np.percentile(entropy, percentile) high_entropy_mask = (entropy > threshold).astype(float) return high_entropy_mask, entropy def analyze_entropy_distribution(self, prompts: List[str], batch_generations: List[List[int]]) -> dict: """ Analyze entropy patterns across multiple samples. Returns statistics useful for setting percentile threshold. """ all_entropy = [] all_masks = [] for prompt, gen_ids in zip(prompts, batch_generations): entropy = self.compute_token_entropy(prompt, gen_ids) all_entropy.append(entropy) entropy_concat = np.concatenate(all_entropy) return { "mean_entropy": float(entropy_concat.mean()), "std_entropy": float(entropy_concat.std()), "percentile_25": float(np.percentile(entropy_concat, 25)), "percentile_75": float(np.percentile(entropy_concat, 75)), "percentile_90": float(np.percentile(entropy_concat, 90)), "high_entropy_fraction": float((entropy_concat > np.percentile(entropy_concat, 75)).mean()) } # Example usage analyzer = TokenEntropyAnalyzer() prompt = "Solve: 2x + 5 = 13. Step-by-step: " generated_text = "Subtract 5 from both sides: 2x = 8. Divide by 2: x = 4." generated_ids = analyzer.tokenizer.encode(generated_text) high_entropy_mask, entropy = analyzer.identify_high_entropy_tokens( prompt, generated_ids, percentile=75 ) print(f"High-entropy tokens: {high_entropy_mask.sum()}/{len(high_entropy_mask)}") print(f"Entropy stats: mean={entropy.mean():.3f}, std={entropy.std():.3f}") ``` Implement selective gradient updates during RL training: ```python import torch.optim as optim class SelectiveGradientRLTrainer: """RL trainer that optimizes only high-entropy tokens.""" def __init__(self, model, analyzer: TokenEntropyAnalyzer, learning_rate: float = 1e-5, entropy_percentile: float = 75): self.model = model self.analyzer = analyzer self.optimizer = optim.AdamW(model.parameters(), lr=learning_rate) self.entropy_percentile = entropy_percentile def compute_masked_loss(self, logits: torch.Tensor, target_ids: torch.Tensor, entropy_mask: torch.Tensor, rewards: torch.Tensor) -> torch.Tensor: """ Compute RL loss only for high-entropy positions. Standard REINFORCE: log_prob * reward, but only for masked positions. """ # Get log probabilities log_probs = F.log_softmax(logits, dim=-1) selected_log_probs = log_probs.gather(-1, target_ids.unsqueeze(-1)).squeeze(-1) # Apply entropy mask: zero out low-entropy positions masked_log_probs = selected_log_probs * entropy_mask # REINFORCE objective: higher reward = higher log prob # Negate because optimizer minimizes loss loss = -(masked_log_probs * rewards.unsqueeze(-1)).mean() return loss def train_step(self, prompts: List[str], generated_ids_list: List[List[int]], rewards: List[float]) -> dict: """ Single RL training step using selective gradient updates. """ batch_size = len(prompts) total_loss = 0 total_high_entropy = 0 total_tokens = 0 for prompt, gen_ids, reward in zip(prompts, generated_ids_list, rewards): # Get entropy mask high_entropy_mask, entropy = self.analyzer.identify_high_entropy_tokens( prompt, gen_ids, percentile=self.entropy_percentile ) total_high_entropy += high_entropy_mask.sum() total_tokens += len(high_entropy_mask) # Prepare inputs full_ids = self.analyzer.tokenizer.encode(prompt + " " + self.analyzer.tokenizer.decode(gen_ids), return_tensors="pt") # Forward pass outputs = self.model(full_ids, output_hidden_states=True) logits = outputs.logits[0, :-1] # Predict all but last target_ids = full_ids[0, 1:] # Target is next token # Compute loss only for high-entropy tokens mask_tensor = torch.tensor(high_entropy_mask, device=logits.device) reward_tensor = torch.tensor([reward] * len(high_entropy_mask), device=logits.device) loss = self.compute_masked_loss(logits, target_ids, mask_tensor, reward_tensor) total_loss += loss.item() # Backward pass loss.backward() # Update parameters self.optimizer.step() self.optimizer.zero_grad() # Return statistics return { "loss": total_loss / batch_size, "high_entropy_fraction": total_high_entropy / total_tokens, "tokens_optimized": int(total_high_entropy) } # Example training loop model = AutoModelForCausalLM.from_pretrained("gpt2-medium") trainer = SelectiveGradientRLTrainer(model, analyzer, entropy_percentile=75) # Mock training data prompts = ["Solve 3x - 7 = 5. Step-by-step: "] * 4 generations = [ [106, 14, 527, 10, 326, 13], # Different solution attempts [106, 14, 527, 10, 326, 13], [106, 14, 527, 10, 326, 13], [106, 14, 527, 10, 326, 13], ] rewards = [1.0, 0.5, 1.0, 0.0] # Correctness signal: 1 = correct, 0 = incorrect for epoch in range(3): stats = trainer.train_step(prompts, generations, rewards) print(f"Epoch {epoch+1}: Loss={stats['loss']:.4f}, " f"High-entropy fraction={stats['high_entropy_fraction']:.1%}") ``` Compare selective updates to baseline full-parameter updates: ```python class ComparisonBenchmark: """Compare selective vs. standard RL training.""" def __init__(self, model_name: str = "gpt2-medium"): self.model_selective = AutoModelForCausalLM.from_pretrained(model_name) self.model_standard = AutoModelForCausalLM.from_pretrained(model_name) self.analyzer = TokenEntropyAnalyzer(model_name) self.trainer_selective = SelectiveGradientRLTrainer( self.model_selective, self.analyzer, entropy_percentile=75 ) self.trainer_standard = StandardRLTrainer(self.model_standard) def run_comparison(self, test_prompts: List[str], test_labels: List[int], num_epochs: int = 10): """Train both models and compare convergence.""" selective_losses = [] standard_losses = [] selective_accuracies = [] standard_accuracies = [] for epoch in range(num_epochs): # Selective training sel_stats = self.trainer_selective.train_step( test_prompts, [self.analyzer.tokenizer.encode(p) for p in test_prompts], [float(l) for l in test_labels] ) selective_losses.append(sel_stats["loss"]) # Standard training (updates all tokens) std_stats = self.trainer_standard.train_step( test_prompts, test_labels ) standard_losses.append(std_stats["loss"]) # Evaluate periodically if epoch % 3 == 0: sel_acc = self.evaluate_accuracy(self.model_selective, test_prompts) std_acc = self.evaluate_accuracy(self.model_standard, test_prompts) selective_accuracies.append(sel_acc) standard_accuracies.append(std_acc) return { "selective_losses": selective_losses, "standard_losses": standard_losses, "selective_accuracies": selective_accuracies, "standard_accuracies": standard_accuracies } def evaluate_accuracy(self, model, prompts: List[str]) -> float: """Simple accuracy evaluation.""" correct = 0 for prompt in prompts: output = model.generate( self.analyzer.tokenizer.encode(prompt, return_tensors="pt"), max_length=50 ) # Placeholder accuracy check correct += 1 # In practice: check against ground truth return correct / len(prompts) # Run benchmark benchmark = ComparisonBenchmark() test_prompts = ["Solve: x + 5 = 10"] * 8 test_labels = [1, 1, 1, 0, 1, 0, 1, 1] results = benchmark.run_comparison(test_prompts, test_labels, num_epochs=10) print(f"Selective approach converges faster: " f"final loss {results['selective_losses'][-1]:.4f} vs " f"{results['standard_losses'][-1]:.4f}") ``` ## Practical Guidance | Aspect | Details | |--------|---------| | **Entropy Percentile** | 75th percentile captures ~25% of tokens; adjust 70-85 based on task complexity | | **Task-Specific Tuning** | Math reasoning: 75-80, Code: 70-75, Language: 85+ (fewer decision points) | | **Compute Savings** | Selective updates reduce gradient compute by ~70-80% per step | | **Convergence Speed** | Typically 2-4x faster convergence vs. standard RL on same dataset | | **Entropy Stability** | Entropy patterns stable across model sizes and checkpoints in practice | **When to Use:** - RL on reasoning tasks with limited compute budget (fewer GPUs, tighter deadline) - Large models where gradient computation dominates training cost - Tasks with verifiable rewards (MATH, code, logic) enabling strong reward signals - Need faster experimentation iteration with RL - Scaling RL to larger batch sizes on fixed hardware **When NOT to Use:** - Tasks where all tokens contribute equally to output quality (poetry, creative writing) - Entropy analysis doesn't align with task structure (domain-specific reasoning) - Extremely low-entropy outputs where selective updates activate too few tokens - Fine-grained stylistic control where repetition patterns matter - Real-time systems with latency constraints (entropy analysis adds overhead) **Common Pitfalls:** - Entropy percentile too aggressive (>85): too few tokens updated, underfitting - Percentile too conservative (<65): minimal compute savings, defeats purpose - Task entropy distribution differs from analysis set: recalibrate on target domain - Reward signal noise: entropy filtering helps but doesn't fix poor reward model - Ignoring task-specific entropy patterns: logic and code have different decision densities ## Reference Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning https://arxiv.org/abs/2506.01939