# Self Hinting Enhance Reinforcement Learning > Apply the SAGE self-hinting technique to improve LLM problem-solving by generating graduated hints that boost solution diversity and prevent reasoning collapse. Use when: 'help me solve this hard math problem step by step', 'generate hints for this coding challenge', 'break down this complex problem with progressive clues', 'design a self-hinting prompt pipeline', 'create a hint curriculum for training data', 'scaffold this reasoning task with decomposition hints'. - Skill: `ndpvt-web/self-hinting-enhance-reinforcement-learning` (Agent Skill) - Install (CLI): `npx skillmds@latest add ndpvt-web/self-hinting-enhance-reinforcement-learning` - Raw SKILL.md: https://api.skillmd.com/api/skills/ndpvt-web/self-hinting-enhance-reinforcement-learning/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: ndpvt-web (https://skillmd.com/u/ndpvt-web) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/ndpvt-web/self-hinting-enhance-reinforcement-learning --- # Self-Hinting for Enhanced LLM Reasoning (SAGE) This skill enables Claude to apply the SAGE (Self-hint Aligned GRPO with privileged supervision) technique from reinforcement learning research to practical problem-solving. The core idea: when a model struggles with a hard problem, generating graduated self-hints — compact plans or decompositions at varying detail levels — dramatically increases the diversity of reasoning paths and prevents the model from getting stuck in a single (often wrong) approach. Claude uses this to scaffold difficult problems, design hint-based prompt pipelines, and build training curricula that adapt to the learner's current skill level. ## When to Use - When a user asks to solve a multi-step math, logic, or coding problem and wants structured reasoning scaffolding - When designing prompt chains that progressively reveal problem structure to an LLM - When building RL training pipelines for LLMs and encountering reward sparsity or advantage collapse - When creating educational content that needs graduated hint systems (Level 1: nudge, Level 2: strategy, Level 3: walkthrough) - When the user wants to break a complex task into a self-hinting decomposition before attempting a solution - When debugging why an LLM consistently fails on certain problem types and you need to reshape its reasoning distribution ## Key Technique **The Problem: Advantage Collapse.** In Group Relative Policy Optimization (GRPO), the model generates G candidate solutions for each prompt and learns from the relative quality differences between them. When a problem is hard, all G rollouts often fail identically — they all score 0. With no variance in rewards, the relative advantage signal collapses to zero, and the model learns nothing from that prompt. This creates a dead zone where the hardest problems (the ones the model most needs to improve on) produce no training signal at all. **The Solution: Self-Generated Hints as Diversity Amplifiers.** SAGE injects a compact hint h (a plan, decomposition, or key insight) between the prompt and the solution during training. The hint is sampled from the model itself at one of three graduated strength levels: Level 1 (minimal nudge — reframe the problem), Level 2 (medium — identify the key technique), Level 3 (detailed — provide a step-by-step skeleton). The critical insight is that the task reward R(x, solution) stays unchanged — the hint only reshapes the distribution of reasoning paths, making it far more likely that at least one of the G rollouts succeeds. This restores the variance needed for GRPO to learn. At test time, the hint is removed entirely (h = empty), and the model solves problems without any privileged information. **The Curriculum Effect.** Because hints are generated by the model itself (not a fixed external source), they naturally track the model's evolving capabilities. As training progresses, fewer prompts need hints — the model learns to internalize the reasoning patterns the hints were scaffolding. This creates an adaptive curriculum: hints concentrate on the model's current bottlenecks, then fade as those bottlenecks are resolved. ## Step-by-Step Workflow 1. **Assess problem difficulty.** Read the problem and classify it: can you solve it directly in one pass, or does it require multi-step reasoning where individual steps could go wrong? Self-hinting is valuable for the latter category. 2. **Generate Level 1 hint (minimal nudge).** Restate the problem in a way that exposes its structure without giving away the approach. Example: "Rewrite the base-b numerals as ordinary integers in terms of b, then express divisibility as a constraint on linear expressions." 3. **Generate Level 2 hint (strategy identification).** Name the specific technique or decomposition needed. Example: "Convert to form Ab+C. If (b+7) divides (9b+7), compute 9b+7 minus multiples of (b+7) to eliminate the b term." 4. **Generate Level 3 hint (detailed skeleton).** Provide a step-by-step procedure that stops short of the actual computation. Example: "Step 1: Compute 17_b = b+7 and 97_b = 9b+7. Step 2: Compute (9b+7) - 9(b+7) = -56. Step 3: Find all divisors of 56 greater than 9+7=16. Step 4: Sum the corresponding b values." 5. **Select the minimum effective hint level.** Start with Level 1. Only escalate if the reasoning path under that hint still collapses to a single failure mode. The goal is the weakest hint that produces diverse solution attempts. 6. **Solve conditioned on the selected hint.** Use the hint as a prefix to your reasoning, generating the full solution with the hint as scaffolding context. 7. **Verify the solution against the original problem.** Apply the terminal reward: does the answer actually satisfy the original constraints? The hint must not change what counts as correct. 8. **Strip the hint from the final output.** Present the clean solution without the scaffolding. The hint was a training wheel — the final reasoning should stand on its own. 9. **If building a training pipeline:** Implement policy-dependent scheduling — only activate hints when group reward variance is zero (all rollouts fail or all succeed). Share one hint realization across all G rollouts per prompt per epoch to reduce variance. 10. **Track hint frequency over time.** If using this iteratively, monitor how often hints are needed. Decreasing hint usage signals the model (or your reasoning) is internalizing the scaffolded patterns. ## Concrete Examples **Example 1: Math Competition Problem** User: "Find the sum of all integer bases b > 9 where 17_b divides 97_b." Approach: 1. **Level 1 hint (internal):** "Express base-b numerals as polynomials in b, then reformulate divisibility." 2. **Level 2 hint (internal):** "17_b = b+7, 97_b = 9b+7. Divisibility means (b+7) | (9b+7). Eliminate b via linear combination." 3. **Solve with Level 2:** Compute (9b+7) - 9(b+7) = 9b+7 - 9b-63 = -56. So (b+7) | 56. Divisors of 56: 1,2,4,7,8,14,28,56. Since b > 9, b+7 > 16, so b+7 in {28, 56}, giving b in {21, 49}. Sum = 70. Output (hint-free): ``` 17 in base b is b+7, and 97 in base b is 9b+7. For (b+7) to divide (9b+7), note that (9b+7) - 9(b+7) = -56. So (b+7) must divide 56. The divisors of 56 exceeding 16 (since b > 9) are 28 and 56, giving b = 21 and b = 49. The sum is 21 + 49 = **70**. ``` **Example 2: Designing a Hint Pipeline for Code Generation** User: "I'm training an LLM to generate Python solutions for LeetCode problems. Many hard problems get 0% pass rate across all samples. How do I apply self-hinting?" Approach: 1. Identify the reward collapse: if all K samples for a prompt fail all test cases, GRPO gets zero gradient. 2. Design three hint levels for code problems: - Level 1: Name the algorithm class ("This is a dynamic programming problem on intervals") - Level 2: Specify the state representation ("Define dp[i][j] as the minimum cost to merge stones from index i to j") - Level 3: Provide the recurrence skeleton ("For each split point k in [i,j-1], dp[i][j] = min(dp[i][k] + dp[k+1][j]) + sum(stones[i..j])") 3. Implement policy-dependent activation: only inject hints when all G rollouts for a prompt fail all test cases. Output: ```python # SAGE-style hint pipeline for code generation training def generate_hint(problem: str, reference_solution: str, level: int) -> str: """Generate a graduated hint from the reference solution.""" if level == 1: prompt = f"""Given this coding problem and its solution, state ONLY the algorithm category in one sentence. Problem: {problem} Solution: {reference_solution} Hint:""" elif level == 2: prompt = f"""Given this coding problem and its solution, describe the key data structure and state representation in 2-3 sentences. Problem: {problem} Solution: {reference_solution} Hint:""" elif level == 3: prompt = f"""Given this coding problem and its solution, provide a step-by-step procedure (no code) covering the core recurrence or logic. Problem: {problem} Solution: {reference_solution} Hint:""" return call_hint_model(prompt) def should_activate_hint(rewards: list[float]) -> bool: """Activate hints only when group variance is zero (advantage collapse).""" return len(set(rewards)) <= 1 def sage_training_step(prompt, reference, model, G=16): # First, try without hints rollouts = [model.generate(prompt) for _ in range(G)] rewards = [evaluate(prompt, r) for r in rollouts] if should_activate_hint(rewards): # Escalate through hint levels until diversity emerges for level in [1, 2, 3]: hint = generate_hint(prompt, reference, level) augmented = f"{prompt}\n\nHint: {hint}" rollouts = [model.generate(augmented) for _ in range(G)] rewards = [evaluate(prompt, r) for r in rollouts] if len(set(rewards)) > 1: break # Diversity restored # Compute GRPO advantages from the (now diverse) reward distribution advantages = compute_grpo_advantages(rewards) return policy_gradient_update(model, rollouts, advantages) ``` **Example 3: Scaffolding a Complex Debugging Task** User: "My distributed system has a race condition that only manifests under high load. I've been staring at the code for hours." Approach: 1. **Level 1 hint:** "Identify all shared mutable state accessed by concurrent handlers. Focus on resources acquired in one goroutine and released in another." 2. **Level 2 hint:** "The race is between the connection pool's checkout and the health-check eviction loop. Both hold a read lock on the pool map but the eviction path upgrades to a write lock mid-operation." 3. **Level 3 hint:** "Trace the lock acquisition order: checkout takes pool.RLock then conn.Lock; eviction takes pool.Lock then conn.Lock. Under high load, checkout holds pool.RLock while waiting for conn.Lock, and eviction is blocked on pool.Lock upgrade. This is a classic lock-ordering deadlock, not a data race." Output (hint-stripped): ``` The issue is a lock-ordering deadlock between connection checkout and health-check eviction. Checkout acquires pool.RLock -> conn.Lock, while eviction acquires pool.Lock -> conn.Lock. Under contention, these create a circular wait. Fix: make eviction collect connections to close under pool.Lock, release pool.Lock, then close them without holding the pool lock. ``` ## Best Practices - **Do:** Start with the weakest hint level that breaks the reasoning deadlock. Over-hinting produces correct answers but doesn't build transferable reasoning patterns. - **Do:** Keep hints structural (plans, decompositions, technique names) rather than computational (intermediate numerical results). Structural hints generalize; computational hints are problem-specific. - **Do:** Verify solutions against the original problem statement, not against the hint. The hint is scaffolding, not ground truth — a hint can point in a useful direction while being slightly imprecise. - **Do:** Share one hint across all rollouts for the same prompt in a training batch. Using different hints per rollout violates Jensen's inequality on the gate probability and reduces expected diversity (Remark 3.4 in the paper). - **Avoid:** Generating hints from a frozen or external model. Self-generated hints track the learner's evolving frontier; external hints may be too easy or too hard. - **Avoid:** Always applying hints regardless of difficulty. Policy-dependent scheduling (hint only when advantage collapses) is critical — unnecessary hints waste capacity and can create hint dependency. ## Error Handling - **Hint produces no diversity improvement:** Escalate to the next hint level. If Level 3 still fails, the problem may be beyond the model's current capability — skip it and revisit after further training. - **Model becomes hint-dependent (test performance drops without hints):** This indicates the hint is leaking answer information rather than scaffolding reasoning. Reduce hint detail level and ensure hints describe *process* not *results*. - **Reward variance is always zero even with hints:** Check the reward function — it may be too coarse (binary pass/fail on multi-part problems). Consider partial credit or decomposing the verification into sub-checks. - **Hints contradict each other across levels:** The hint generator has inconsistent understanding of the problem. Re-derive hints from the reference solution with explicit level constraints rather than generating them independently. ## Limitations - Self-hinting requires access to a reference solution (or verifiable reward) during training to generate meaningful hints. It cannot operate in purely unsupervised settings. - The technique is designed for problems with **verifiable terminal rewards** (math, code, logic). It does not directly apply to open-ended creative tasks where correctness is subjective. - At test time, the model must perform without hints. If the gap between hint-conditioned and no-hint performance is large, the training may not be transferring the scaffolded reasoning patterns effectively. - Hint generation adds computational overhead during training (though SAGE-light mitigates this with epoch-level thresholds instead of per-prompt probing). - The optimal hint strength (targeting ~50% success rate within a group) assumes enough rollout budget G to observe the variance. Very small G values reduce the technique's effectiveness. ## Reference **Paper:** [Self-Hinting Language Models Enhance Reinforcement Learning](https://arxiv.org/abs/2602.03143v1) (Liao et al., 2026). Focus on Section 3 for the mathematical analysis of advantage collapse and the gate-opening probability, Algorithm 1 for the training loop, and Section 3.3 for why single-hint-per-group outperforms multi-hint sampling. Code: [github.com/BaohaoLiao/SAGE](https://github.com/BaohaoLiao/SAGE).