adu2021 Memex(RL): Indexed Experience Memory for Scaling Long-Horizon Agents
Long-horizon multi-step reasoning tasks overwhelm finite context windows: agents either repeat full conversation history (inefficient) or use lossy summaries (information loss). Memex introduces a memory architecture where agents explicitly manage both working context and long-term storage through learned operations. The system maintains compact in-context indexed summaries with pointers to externally archived artifacts, enabling full-fidelity evidence retrieval on demand.
The core innovation treats memory as a first-class agent capability: read operations dereference indices to retrieve exact past information; write operations compress state into indexed summaries. This mirrors how humans manage complex projects with external notes and references while keeping working memory focused.
Core Concept
Memex decomposes memory into two complementary systems:
- Working Context: Compact in-context summary (~1-2KB) containing indexed references to past interactions
- Experience Store: External archive of full-fidelity artifacts (full trajectories, tool outputs, decision trees) under stable indices
Agents learn to:
- Write: Compress key information into indexed summary for future reference
- Read: Dereference indices to retrieve exact past information when needed
- Manage: Decide what to compress, when to retrieve, based on task demands
This separation enables agents to maintain reasoning coherence over thousands of tokens while staying within practical context limits.
Architecture Overview
- Input: Long-horizon task with multi-step subtasks
- Working Context Module: Compressed summary with embedded indices
- Experience Store: Persistent key-value archive indexed by discrete IDs
- Read/Write Operators: Agent-learned memory operations
- Action Space: Extended with memory operations (read_idx, write_summary)
- Output: Decision with access to complete historical context
Implementation Steps
Step 1: Design experience store and indexing scheme
Create a persistent storage system for archived experiences with stable access keys.
class ExperienceStore:
"""External memory store for agent artifacts."""
def __init__(self, max_size=10000):
self.store = {}
self.index_counter = 0
self.max_size = max_size
def write(self, artifact, artifact_type='trajectory'):
"""
Archive an artifact and return stable index.
artifact_type: 'trajectory', 'tool_output', 'decision_tree', etc.
"""
if len(self.store) >= self.max_size:
# Evict least-recently-used entry
oldest_idx = min(self.store.keys(),
key=lambda k: self.store[k]['access_time'])
del self.store[oldest_idx]
idx = self.index_counter
self.store[idx] = {
'artifact': artifact,
'type': artifact_type,
'timestamp': time.time(),
'access_time': time.time(),
'access_count': 0
}
self.index_counter += 1
return idx
def read(self, idx):
"""Retrieve archived artifact by index."""
if idx not in self.store:
return None
entry = self.store[idx]
entry['access_time'] = time.time()
entry['access_count'] += 1
return entry['artifact']
def list_indices(self):
"""Return available indices for agent querying."""
return sorted(self.store.keys())
Step 2: Create working context with indexed summaries
Design a compact in-context representation that embeds indices without storing full artifacts.
def create_indexed_summary(current_trajectory, experience_indices,
summary_max_tokens=200):
"""
Create compact working context with embedded index references.
Format:
- Current step context (80-100 tokens)
- Index references to archived experiences (20-40 tokens)
- Metadata about available archived artifacts (remaining tokens)
"""
# Part 1: Recent context (last 2-3 steps)
recent_context = format_recent_trajectory(current_trajectory, depth=3)
# Part 2: Archived experience index pointers
index_pointers = []
for idx in experience_indices[-5:]: # Reference last 5 archived experiences
summary = f"[exp_{idx}] " # Compact reference
index_pointers.append(summary)
# Part 3: Metadata about available memories
available_memories = f"Available: {len(experience_indices)} archived experiences"
# Combine into compact summary
working_context = f"""
Current Context:
{recent_context}
Available Memories:
{' '.join(index_pointers)}
{available_memories}
Agent Action Space: [act, read_exp_IDX, write_summary]
"""
return working_context.strip()
def format_recent_trajectory(trajectory, depth=3):
"""Format last `depth` steps of trajectory for context."""
recent = trajectory[-depth:]
formatted = []
for step in recent:
formatted.append(f"Step: {step['action']}, Result: {step['result'][:50]}")
return '\n'.join(formatted)
Step 3: Implement read/write agent actions
Extend agent action space to include memory operations as learned behaviors.
class MemoryAugmentedAgent:
"""LLM agent with learned memory operations."""
def __init__(self, llm_model, experience_store, working_context_size=200):
self.llm = llm_model
self.store = experience_store
self.working_context_size = working_context_size
def step(self, task_prompt, working_memory, trajectory):
"""
Execute agent step with optional memory operations.
Returns: (action, memory_operation)
"""
# Construct full prompt with working memory
indexed_summary = create_indexed_summary(trajectory, self.store.list_indices())
full_prompt = f"""
Task: {task_prompt}
Working Memory:
{indexed_summary}
Your action (format: ACTION:value or READ:exp_IDX or WRITE:summary_text):
"""
# Forward pass through LLM
response = self.llm.generate(full_prompt, max_tokens=100)
# Parse response for action type
if response.startswith('READ:'):
# Memory read operation
exp_idx = int(response.split(':')[1])
artifact = self.store.read(exp_idx)
return ('read', artifact, exp_idx)
elif response.startswith('WRITE:'):
# Memory write operation
summary = response.split(':', 1)[1]
idx = self.store.write(summary, artifact_type='agent_summary')
return ('write', idx)
else:
# Regular action
return ('act', response)
def compress_for_storage(self, trajectory_segment):
"""
Compress trajectory segment into archived summary.
Executed when agent issues WRITE action.
"""
# Summarize key decisions and outcomes
summary = f"""
Trajectory Summary:
- Steps: {len(trajectory_segment)}
- Key decisions: {[s['action'] for s in trajectory_segment[:3]]}
- Final outcome: {trajectory_segment[-1]['result']}
"""
return summary
Step 4: Learn memory operations via reinforcement learning
Train agent to optimize when and what to read/write using RL.
def compute_memory_cost(operation_type, access_count=0):
"""
Compute cost of memory operation in tokens.
- read: 1-2 tokens (index) + retrieved artifact tokens
- write: summary length + index (2-3 tokens)
"""
if operation_type == 'read':
# Cost: index overhead + artifact retrieval
return 2 + access_count # Penalize repeated reads of same info
elif operation_type == 'write':
# Cost: summary compression + indexing overhead
return 3 + 0 # Fixed overhead
else:
return 0 # No memory cost for regular actions
def compute_memory_reward(trajectory, task_success):
"""
Reward successful task completion while penalizing memory operations.
reward = task_success_bonus - memory_operation_cost
"""
success_bonus = 10.0 if task_success else -1.0
# Count memory operations in trajectory
total_memory_cost = 0
for step in trajectory:
if step['action_type'] in ['read', 'write']:
total_memory_cost += compute_memory_cost(step['action_type'])
# Memory efficiency reward
memory_penalty = 0.1 * total_memory_cost
return success_bonus - memory_penalty
def memory_rl_update(agent, batch_trajectories, learning_rate=0.001):
"""
RL update to optimize memory operations.
Uses policy gradient with memory cost penalties.
"""
total_pg_loss = 0.0
for trajectory in batch_trajectories:
task_success = trajectory[-1]['task_complete']
reward = compute_memory_reward(trajectory, task_success)
# Policy gradient for each step
for t, step in enumerate(trajectory):
# Log probability of chosen action
logprob = agent.llm.compute_logprob(
step['prompt'],
step['action']
)
# PG loss: maximize log-prob of low-cost, high-reward actions
step_memory_cost = compute_memory_cost(step['action_type'])
adjusted_reward = reward - step_memory_cost
pg_loss = -logprob * adjusted_reward
total_pg_loss += pg_loss
# Optimize
agent.llm.backward(total_pg_loss / len(batch_trajectories))
agent.llm.optimizer.step(learning_rate)
return (total_pg_loss / len(batch_trajectories)).item()
Step 5: Scaling evaluation on long-horizon tasks
Benchmark memory-augmented agents on tasks requiring hundreds of steps.
def evaluate_scaling(agent, task_sequence, max_steps=500):
"""
Evaluate agent on long-horizon task.
Compare: working context size, success rate, memory operations.
"""
trajectory = []
working_memory = []
context_size = 0
success = False
for step_idx in range(max_steps):
# Agent step
action, memory_op = agent.step(
task_sequence[step_idx],
working_memory,
trajectory
)
trajectory.append({
'step': step_idx,
'action': action,
'memory_op': memory_op,
'context_size': context_size
})
# Update working memory
if memory_op[0] == 'write':
working_memory.append(memory_op[1])
context_size += len(str(memory_op[1]))
# Check task completion
if agent.check_success(action):
success = True
break
return {
'success': success,
'steps_taken': len(trajectory),
'peak_context_size': max(s['context_size'] for s in trajectory),
'memory_ops': len([t for t in trajectory if t['memory_op'] is not None]),
'trajectory': trajectory
}
Practical Guidance
Hyperparameter Selection:
- Working context size: 200-500 tokens. Balance information density vs. token overhead.
- Experience store max size: 1,000-10,000. Larger = more history available; higher eviction cost.
- Memory operation cost weight: 0.1-0.5. Higher = penalizes reads/writes more, encouraging efficient memory usage.
- Access count penalty: 0.1-0.5 per read. Discourages repeated reads of same information; encourages compression.
When to Use:
- Long-horizon multi-step reasoning (100+ steps)
- Tasks requiring history reference (e.g., negotiation, project planning)
- Scenarios where lossy compression is insufficient
- Agent systems with token budget constraints
When NOT to Use:
- Short tasks (<20 steps) where context window is not a bottleneck
- Real-time systems requiring minimal latency (memory operations add overhead)
- Tasks where all information is state-encoded (no explicit history dependency)
Common Pitfalls:
- Memory leakage: Archived summaries can drift from true history. Validate by spot-checking retrieved artifacts.
- Access patterns: Skewed access patterns can cause eviction of useful memories. Use LRU or frequency-based retention.
- Compression degradation: Over-compressed summaries lose critical details. Monitor task performance; increase context size if needed.
- Index fragmentation: After many reads/writes, indices become sparse. Periodic compaction of store helps.