adu2021 Technique: Dual Intrinsic Feedback via Hindsight Self-Reflection
Traditional RL agents rely on sparse binary rewards (task success/failure), limiting learning signals for long-horizon reasoning. RetroAgent augments this with dual intrinsic feedback: capability-evolution rewards that measure incremental progress, plus natural-language lessons extracted from trajectories and reused strategically via memory.
This approach shifts optimization from isolated problem-solving toward continuous agent evolution, enabling agents to learn from partial successes and build persistent knowledge.
Core Concept
RetroAgent generates two complementary feedback signals:
Intrinsic Numerical Feedback: A scalar reward measuring progress toward subtask completion relative to historical baselines, enabling exploration of partial achievements not yet constituting full success.
Intrinsic Language Feedback: Natural-language lessons distilled from trajectories and stored in memory, retrieved during subsequent training via a principled strategy balancing relevance, utility, and coverage.
The hindsight mechanism conditions on successful outcomes, then computes how step probability changes given success, effectively performing causal filtering of responsible actions.
Architecture Overview
- Reflection module: Generates capabilities-evolution rewards from trajectory analysis
- Lesson extraction: Distills actionable insights from successful and near-successful trajectories
- Memory buffer: Persistent storage of lessons with metadata (semantic embedding, utility score, coverage tag)
- SimUtil-UCB retriever: Balances semantic similarity, historical utility, and exploration coverage
- In-context or RL reflection: Optional variants for higher quality or joint optimization
Implementation Steps
Step 1: Compute Capability-Evolution Rewards
Generate numerical feedback based on incremental progress toward task completion.
import torch
from collections import defaultdict
class CapabilityEvolutionReward:
def __init__(self):
self.subtask_progress = defaultdict(float)
def compute_reward(self, trajectory, task_subtasks):
"""
trajectory: list of (action, state) pairs
task_subtasks: list of subtask descriptions
"""
completed_subtasks = set()
for subtask in task_subtasks:
# Check if trajectory addresses this subtask
# (simplified; real implementation uses semantic matching)
if self.trajectory_addresses_subtask(trajectory, subtask):
completed_subtasks.add(subtask)
# Baseline: historical max completion for this task
historical_max = self.subtask_progress[task_id]
current_completion = len(completed_subtasks) / len(task_subtasks)
# Capability-evolution reward: progress relative to history
intrinsic_reward = max(0, current_completion - historical_max)
self.subtask_progress[task_id] = max(historical_max, current_completion)
return intrinsic_reward
def trajectory_addresses_subtask(self, trajectory, subtask):
# In practice, use semantic matching with MLLM
return True # Placeholder
Step 2: Extract and Store Lessons
Distill natural-language lessons from trajectories and populate memory.
class LessonMemory:
def __init__(self, capacity=10000, embedding_model=None):
self.capacity = capacity
self.lessons = []
self.embedding_model = embedding_model
self.utility_scores = []
self.access_counts = defaultdict(int)
def add_lesson(self, trajectory, outcome, lesson_text):
"""Store a lesson with metadata."""
lesson_entry = {
'text': lesson_text,
'trajectory': trajectory,
'outcome': outcome,
'embedding': self.embedding_model.encode(lesson_text),
'utility': 1.0, # Initial utility
'timestamp': time.time(),
'coverage_tag': self.compute_coverage_tag(lesson_text)
}
if len(self.lessons) >= self.capacity:
# Evict oldest lesson
self.lessons.pop(0)
else:
self.lessons.append(lesson_entry)
def compute_coverage_tag(self, lesson_text):
"""Assign coverage tag (e.g., 'arithmetic', 'reasoning')."""
# Simplified; use semantic clustering in practice
return 'general'
Step 3: SimUtil-UCB Lesson Retrieval
Balance semantic relevance, historical utility, and exploration coverage.
import numpy as np
def retrieve_lessons_simutil_ucb(memory, query_text, k=3, exploration_bonus=0.1):
"""
Retrieve lessons using SimUtil-UCB strategy.
exploration_bonus: UCB weight for under-explored lessons
"""
query_embedding = memory.embedding_model.encode(query_text)
scores = []
for i, lesson in enumerate(memory.lessons):
# Similarity: cosine distance to query
similarity = np.dot(
query_embedding / (np.linalg.norm(query_embedding) + 1e-8),
lesson['embedding'] / (np.linalg.norm(lesson['embedding']) + 1e-8)
)
# Utility: exponential moving average of historical benefit
utility = lesson['utility']
# UCB exploration bonus: fewer accesses = higher bonus
access_count = memory.access_counts[i]
ucb_bonus = exploration_bonus / (access_count + 1)
# Combined score
combined_score = similarity + utility + ucb_bonus
scores.append(combined_score)
# Top-k retrieval
top_indices = np.argsort(scores)[-k:][::-1]
retrieved_lessons = [memory.lessons[i] for i in top_indices]
# Update access counts for UCB
for i in top_indices:
memory.access_counts[i] += 1
return retrieved_lessons
Step 4: In-Context Reflection for Higher Quality
Use trajectory pairs to generate lessons via in-context comparison.
def in_context_reflection(model, success_trajectory, near_miss_trajectory):
"""
Compare trajectories to extract actionable lessons.
"""
prompt = f"""
Successful trajectory:
{format_trajectory(success_trajectory)}
Near-miss trajectory:
{format_trajectory(near_miss_trajectory)}
What key decision or reasoning pattern made the difference?
Extract a concise lesson (1-2 sentences) that future attempts should follow.
"""
lesson = model.generate(prompt, max_tokens=100)
return lesson.strip()
Practical Guidance
When to Use:
- Long-horizon reasoning tasks where binary rewards are too sparse
- Multi-step problems requiring persistent knowledge across episodes
- Scenarios where agents benefit from learning from partial successes
- Interactive task synthesis environments
When NOT to Use:
- Short-horizon problems with immediate, clear rewards
- Extremely budget-constrained inference (memory overhead)
- Domains where lessons don't transfer across tasks
Hyperparameter Tuning:
- Lesson memory capacity: 1000-10000 depending on task diversity
- Exploration bonus weight: 0.1-1.0 determines coverage emphasis
- Utility update rate: Use exponential moving average with α=0.9
- Lesson extraction frequency: Every 2-5 successes balances coverage and redundancy
Common Pitfalls:
- Lesson memory bloat causing retrieval slowdown (implement TTL or importance-based eviction)
- Overly generic lessons that don't transfer (enforce specificity in extraction)
- Under-exploration of coverage space if UCB bonus too weak
- Insufficient trajectory diversity in early training (warm-up phase helpful)