# Retroagent Dual Intrinsic Feedback > Train LLM agents to evolve by generating dual intrinsic feedback signals—numerical rewards for capability progress and language lessons for future reuse. Combines hindsight reflection with memory-augmented RL using SimUtil-UCB retrieval. - Skill: `adu2021/retroagent-dual-intrinsic-feedback` (Agent Skill) - Install (CLI): `npx skillmds@latest add adu2021/retroagent-dual-intrinsic-feedback` - Raw SKILL.md: https://api.skillmd.com/api/skills/adu2021/retroagent-dual-intrinsic-feedback/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - License: MIT - Author: adu2021 (https://skillmd.com/u/adu2021) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/adu2021/retroagent-dual-intrinsic-feedback --- # Technique: Dual Intrinsic Feedback via Hindsight Self-Reflection Traditional RL agents rely on sparse binary rewards (task success/failure), limiting learning signals for long-horizon reasoning. RetroAgent augments this with **dual intrinsic feedback**: capability-evolution rewards that measure incremental progress, plus natural-language lessons extracted from trajectories and reused strategically via memory. This approach shifts optimization from isolated problem-solving toward continuous agent evolution, enabling agents to learn from partial successes and build persistent knowledge. ## Core Concept RetroAgent generates two complementary feedback signals: 1. **Intrinsic Numerical Feedback**: A scalar reward measuring progress toward subtask completion relative to historical baselines, enabling exploration of partial achievements not yet constituting full success. 2. **Intrinsic Language Feedback**: Natural-language lessons distilled from trajectories and stored in memory, retrieved during subsequent training via a principled strategy balancing relevance, utility, and coverage. The hindsight mechanism conditions on successful outcomes, then computes how step probability changes given success, effectively performing causal filtering of responsible actions. ## Architecture Overview - **Reflection module**: Generates capabilities-evolution rewards from trajectory analysis - **Lesson extraction**: Distills actionable insights from successful and near-successful trajectories - **Memory buffer**: Persistent storage of lessons with metadata (semantic embedding, utility score, coverage tag) - **SimUtil-UCB retriever**: Balances semantic similarity, historical utility, and exploration coverage - **In-context or RL reflection**: Optional variants for higher quality or joint optimization ## Implementation Steps ### Step 1: Compute Capability-Evolution Rewards Generate numerical feedback based on incremental progress toward task completion. ```python import torch from collections import defaultdict class CapabilityEvolutionReward: def __init__(self): self.subtask_progress = defaultdict(float) def compute_reward(self, trajectory, task_subtasks): """ trajectory: list of (action, state) pairs task_subtasks: list of subtask descriptions """ completed_subtasks = set() for subtask in task_subtasks: # Check if trajectory addresses this subtask # (simplified; real implementation uses semantic matching) if self.trajectory_addresses_subtask(trajectory, subtask): completed_subtasks.add(subtask) # Baseline: historical max completion for this task historical_max = self.subtask_progress[task_id] current_completion = len(completed_subtasks) / len(task_subtasks) # Capability-evolution reward: progress relative to history intrinsic_reward = max(0, current_completion - historical_max) self.subtask_progress[task_id] = max(historical_max, current_completion) return intrinsic_reward def trajectory_addresses_subtask(self, trajectory, subtask): # In practice, use semantic matching with MLLM return True # Placeholder ``` ### Step 2: Extract and Store Lessons Distill natural-language lessons from trajectories and populate memory. ```python class LessonMemory: def __init__(self, capacity=10000, embedding_model=None): self.capacity = capacity self.lessons = [] self.embedding_model = embedding_model self.utility_scores = [] self.access_counts = defaultdict(int) def add_lesson(self, trajectory, outcome, lesson_text): """Store a lesson with metadata.""" lesson_entry = { 'text': lesson_text, 'trajectory': trajectory, 'outcome': outcome, 'embedding': self.embedding_model.encode(lesson_text), 'utility': 1.0, # Initial utility 'timestamp': time.time(), 'coverage_tag': self.compute_coverage_tag(lesson_text) } if len(self.lessons) >= self.capacity: # Evict oldest lesson self.lessons.pop(0) else: self.lessons.append(lesson_entry) def compute_coverage_tag(self, lesson_text): """Assign coverage tag (e.g., 'arithmetic', 'reasoning').""" # Simplified; use semantic clustering in practice return 'general' ``` ### Step 3: SimUtil-UCB Lesson Retrieval Balance semantic relevance, historical utility, and exploration coverage. ```python import numpy as np def retrieve_lessons_simutil_ucb(memory, query_text, k=3, exploration_bonus=0.1): """ Retrieve lessons using SimUtil-UCB strategy. exploration_bonus: UCB weight for under-explored lessons """ query_embedding = memory.embedding_model.encode(query_text) scores = [] for i, lesson in enumerate(memory.lessons): # Similarity: cosine distance to query similarity = np.dot( query_embedding / (np.linalg.norm(query_embedding) + 1e-8), lesson['embedding'] / (np.linalg.norm(lesson['embedding']) + 1e-8) ) # Utility: exponential moving average of historical benefit utility = lesson['utility'] # UCB exploration bonus: fewer accesses = higher bonus access_count = memory.access_counts[i] ucb_bonus = exploration_bonus / (access_count + 1) # Combined score combined_score = similarity + utility + ucb_bonus scores.append(combined_score) # Top-k retrieval top_indices = np.argsort(scores)[-k:][::-1] retrieved_lessons = [memory.lessons[i] for i in top_indices] # Update access counts for UCB for i in top_indices: memory.access_counts[i] += 1 return retrieved_lessons ``` ### Step 4: In-Context Reflection for Higher Quality Use trajectory pairs to generate lessons via in-context comparison. ```python def in_context_reflection(model, success_trajectory, near_miss_trajectory): """ Compare trajectories to extract actionable lessons. """ prompt = f""" Successful trajectory: {format_trajectory(success_trajectory)} Near-miss trajectory: {format_trajectory(near_miss_trajectory)} What key decision or reasoning pattern made the difference? Extract a concise lesson (1-2 sentences) that future attempts should follow. """ lesson = model.generate(prompt, max_tokens=100) return lesson.strip() ``` ## Practical Guidance **When to Use:** - Long-horizon reasoning tasks where binary rewards are too sparse - Multi-step problems requiring persistent knowledge across episodes - Scenarios where agents benefit from learning from partial successes - Interactive task synthesis environments **When NOT to Use:** - Short-horizon problems with immediate, clear rewards - Extremely budget-constrained inference (memory overhead) - Domains where lessons don't transfer across tasks **Hyperparameter Tuning:** - **Lesson memory capacity**: 1000-10000 depending on task diversity - **Exploration bonus weight**: 0.1-1.0 determines coverage emphasis - **Utility update rate**: Use exponential moving average with α=0.9 - **Lesson extraction frequency**: Every 2-5 successes balances coverage and redundancy **Common Pitfalls:** - Lesson memory bloat causing retrieval slowdown (implement TTL or importance-based eviction) - Overly generic lessons that don't transfer (enforce specificity in extraction) - Under-exploration of coverage space if UCB bonus too weak - Insufficient trajectory diversity in early training (warm-up phase helpful) ## Reference [RetroAgent paper on arXiv](https://arxiv.org/abs/2603.08561)