# Ifdecorator Instruction Following Rl > Enhances RLVR through cooperative-adversarial flywheel, intent verification, and trap instructions. Detects reward hacking and improves training efficiency, achieving 87.43% on IFEval. - Skill: `adu2021/ifdecorator-instruction-following-rl` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/ifdecorator-instruction-following-rl` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/ifdecorator-instruction-following-rl/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/ifdecorator-instruction-following-rl --- # IFDecorator: Instruction Following RL with Verifiable Rewards ## Core Concept IFDecorator strengthens Reinforcement Learning with Verifiable Rewards (RLVR) by addressing two critical challenges: training inefficiency from inadequate difficulty assessment, and reward hacking where models exploit verification shortcuts without aligning to true user intent. The framework uses a cooperative-adversarial flywheel to progressively increase task difficulty, enforces alignment through intent checking, and detects shortcut exploitation through trap instructions. ## Architecture Overview - **Cooperative-Adversarial Flywheel**: Co-evolves instructions and verifications to generate progressively challenging pairs - **IntentCheck Module**: Verifies that model outputs align with actual user intent - **Trip Wires**: Diagnostic system using trap instructions to detect reward hacking - **Sample Efficiency**: Systematically increases task difficulty rather than random sampling - **Robustness**: Detects and prevents exploitation of verification shortcuts ## Implementation Steps ### Step 1: Build the Cooperative-Adversarial Flywheel Create a system that co-evolves instruction-verification pairs with increasing difficulty. ```python class CooperativeAdversarialFlywheel: """ Co-evolve instructions and verifications for progressive difficulty. """ def __init__(self, generator_model, verifier_model, initial_difficulty=1.0): self.generator = generator_model self.verifier = verifier_model self.difficulty = initial_difficulty self.pairs = [] def generate_instruction_pair(self, topic, difficulty_level): """ Generate instruction-verification pair at target difficulty. Args: topic: Task topic or category difficulty_level: Target difficulty (1.0 = easy, 10.0 = hard) Returns: (instruction, verification) tuple """ prompt = f""" Generate a challenging instruction for task: {topic} Difficulty level: {difficulty_level}/10.0 The instruction should: 1. Be clear and unambiguous 2. Require non-trivial following to succeed 3. Have measurable completion criteria Return as JSON with "instruction" and "completion_criteria" fields. """ instruction_pair = self.generator.generate(prompt) return instruction_pair def generate_verification(self, instruction, completion_criteria): """ Generate verifier function for instruction. Args: instruction: Task instruction completion_criteria: Criteria for successful completion Returns: Verifier function """ prompt = f""" Create a verification function for: Instruction: {instruction} Criteria: {completion_criteria} Return Python code for a function that: - Takes model output as input - Returns True if instruction followed correctly - Returns False if any criteria not met Make the verification robust to minor variations. """ verifier_code = self.generator.generate(prompt) return self._compile_verifier(verifier_code) def update_difficulty(self, model_success_rate): """ Adapt difficulty based on model performance. Args: model_success_rate: Proportion of tasks successfully completed """ # If model succeeds >80%, increase difficulty if model_success_rate > 0.8: self.difficulty = min(10.0, self.difficulty * 1.2) # If model succeeds <20%, decrease difficulty elif model_success_rate < 0.2: self.difficulty = max(1.0, self.difficulty / 1.2) def _compile_verifier(self, verifier_code): """Safely compile and return verifier function.""" import types # Execute in restricted namespace exec_globals = {"True": True, "False": False} exec(verifier_code, exec_globals) return exec_globals.get("verify_instruction") ``` ### Step 2: Implement IntentCheck Module Create a module that ensures outputs align with true user intent, not just verification criteria. ```python class IntentCheckModule: """ Verify alignment with user intent beyond mechanical verification. """ def __init__(self, intent_model, divergence_threshold=0.3): self.intent_model = intent_model self.divergence_threshold = divergence_threshold def check_intent_alignment(self, instruction, user_intent, model_output): """ Check if model output aligns with true user intent. Args: instruction: Original instruction user_intent: Deep explanation of what user actually wants model_output: Model's generated response Returns: (aligned, alignment_score, reasoning) """ # Extract intent from instruction prompt = f""" Analyze whether this output truly fulfills the user's intent: User Intent: {user_intent} Model Output: {model_output} Consider: 1. Does output serve the underlying goal? 2. Would the user be satisfied? 3. Are there shortcuts that exploit verification but miss intent? Return JSON with: - "aligned": true/false - "score": 0.0-1.0 - "reasoning": brief explanation """ result = self.intent_model.generate(prompt) alignment = result["aligned"] score = result["score"] return alignment, score, result.get("reasoning", "") def detect_intent_divergence(self, outputs_batch, intents_batch): """ Detect systematic divergence between outputs and intents. Args: outputs_batch: Batch of model outputs intents_batch: Corresponding user intents Returns: List of (output_idx, divergence_score) tuples """ divergences = [] for idx, (output, intent) in enumerate(zip(outputs_batch, intents_batch)): aligned, score, _ = self.check_intent_alignment("", intent, output) if score < self.divergence_threshold: divergences.append((idx, score)) return divergences ``` ### Step 3: Implement Trip Wires for Reward Hacking Detection Create diagnostic traps that detect models exploiting shortcuts. ```python class TripWireDetector: """ Diagnostic system using trap instructions to detect reward hacking. """ def __init__(self): self.trap_instructions = [] self.hacking_indicators = {} def create_trap_instruction(self, base_instruction, exploit_path): """ Create instruction that model should handle correctly. Args: base_instruction: Original instruction exploit_path: Potential shortcut/hack the model might use Returns: Trap instruction that exposes the shortcut """ # Create instruction where the exploit would fail trap = f""" {base_instruction} IMPORTANT: The following will NOT work: - {exploit_path} You must solve this correctly without the shortcut. """ self.trap_instructions.append({ "base": base_instruction, "exploit": exploit_path, "trap": trap }) return trap def test_for_hacking(self, model, instruction, expected_output): """ Test if model is using exploits or solving genuinely. Args: model: Model to test instruction: Original instruction expected_output: Correct output Returns: (is_hacking, confidence, evidence) """ # Generate output on original original_output = model.generate(instruction) # Generate output on trap version trap_instruction = self.create_trap_instruction(instruction, "unknown") trap_output = model.generate(trap_instruction) # Compare outputs if self._similar_patterns(original_output, trap_output): # Model likely using shortcut return True, 0.8, "Similar output on trap instruction" else: # Model adapted behavior return False, 0.8, "Different behavior on trap" def _similar_patterns(self, output1, output2): """Check if outputs follow similar patterns (hacking indicator).""" from difflib import SequenceMatcher ratio = SequenceMatcher(None, output1, output2).ratio() return ratio > 0.7 def create_exploit_detection_set(self): """ Create comprehensive set of trap instructions. Returns: Structured set of traps for model evaluation """ common_exploits = [ "verbosity without substance", "format compliance without content", "keyword stuffing", "logical shortcuts in reasoning", "copying from examples without adaptation" ] traps = [] for exploit in common_exploits: trap = { "exploit_type": exploit, "trap_instruction": self._generate_trap_for_exploit(exploit), "success_indicator": self._get_success_indicator(exploit) } traps.append(trap) return traps def _generate_trap_for_exploit(self, exploit_type): """Generate trap for specific exploit type.""" traps = { "verbosity": "Provide answer in single sentence", "format_compliance": "Answer must violate format constraints", "keyword_stuffing": "Answer cannot contain common keywords" } return traps.get(exploit_type, "") def _get_success_indicator(self, exploit_type): """Get indicator of successful mitigation.""" return lambda output: len(output) < 100 ``` ### Step 4: Integrate into RLVR Training Loop Combine all components into cohesive RLVR training system. ```python def train_instruction_following_with_ifdecorator( model, initial_instructions, config ): """ Train model with IFDecorator framework. Args: model: LLM to train initial_instructions: Starting instruction set config: Training configuration Returns: Trained model with improved instruction following """ flywheel = CooperativeAdversarialFlywheel( model, verifier_model=load_verifier(), initial_difficulty=config.initial_difficulty ) intent_checker = IntentCheckModule() trap_detector = TripWireDetector() # Training loop for episode in range(config.num_episodes): # 1. Generate progressive difficulty examples instruction, verification = flywheel.generate_instruction_pair( topic="instruction_following", difficulty_level=flywheel.difficulty ) # 2. Get model response model_output = model.generate(instruction) # 3. Check verification verified = verification(model_output) # 4. Check intent alignment aligned, intent_score, _ = intent_checker.check_intent_alignment( instruction, user_intent="Genuine task completion", model_output=model_output ) # 5. Detect reward hacking is_hacking, hack_confidence, _ = trap_detector.test_for_hacking( model, instruction, expected_output="correct" ) # Compute reward reward = 0.0 if verified: reward += config.verification_reward if aligned: reward += config.intent_reward if not is_hacking: reward += config.genuine_solving_reward else: reward -= config.hacking_penalty # 6. RL update model.update_from_reward(instruction, model_output, reward) # 7. Update difficulty if episode % config.difficulty_update_freq == 0: success_rate = calculate_recent_success_rate(model) flywheel.update_difficulty(success_rate) return model ``` ## Practical Guidance ### When to Use IFDecorator - **Instruction following benchmarks**: IFEval, MTEval where intent matters - **Complex multi-step tasks**: Where shortcuts exist but don't solve genuine problems - **Safety-critical systems**: Where reward hacking has real consequences - **Dynamic task difficulty**: Where progressive training improves sample efficiency ### When NOT to Use IFDecorator - **Simple verification tasks**: Where verification and intent perfectly align - **Low-risk applications**: Reward hacking mitigation adds computational cost - **Real-time training**: Trap instruction generation and testing add latency - **Insufficient intent labels**: Framework requires explicit intent statements ### Hyperparameter Recommendations - **Initial difficulty**: 2.0-3.0 on 1-10 scale - **Verification reward**: 0.5 (baseline) - **Intent reward**: 0.3-0.5 (emphasize intent) - **Hacking penalty**: 1.0-2.0 (strong negative signal) - **Difficulty update frequency**: Every 20-50 episodes ### Key Insights The key innovation is explicit detection and prevention of reward hacking through trip wires. By creating instructions where common exploits fail, the system forces genuine problem-solving. The cooperative-adversarial flywheel ensures efficient training by focusing on task-specific difficulty, not random sampling. ## Reference **IFDecorator: Instruction Following RL with Verifiable Rewards** (arXiv:2508.04632) Introduces cooperative-adversarial flywheel, IntentCheck module, and trip wire detector system for robust RLVR. Achieves 87.43% on IFEval by preventing reward hacking while improving training efficiency.