Memory Complete

Complete Memory System Skill

A production-ready memory system for persistent AI agent context: semantic vectors + compaction + extraction rules.

Overview

This skill provides a complete, production-ready memory system that you can:

1. Clone from GitHub → Get entire configuration ready
2. Run setup.py → Initialize in 30 seconds
3. Start using → No additional configuration needed (already optimized)
4. Customize later → Adapt rules/agents to your specific needs

Everything is pre-configured based on real production experience.

⚡ Quick Start (3 Steps)

Step 1: Clone or Copy

# Option A: Git clone
git clone https://github.com/YOUR_USERNAME/memory-complete.git
cd memory-complete

# Option B: Copy entire folder
cp -r /path/to/memory-complete ./my-project/
cd my-project

Step 2: Install & Setup

# Install dependencies
pip install -r requirements.txt

# Initialize system (one-time)
python3 code/setup.py

# Expected output:
# ✓ Memory system initialized
# ✓ Vector database ready
# ✓ Configuration loaded
# ✓ Ready to use

Step 3: Start Using

from openclaw_bridge_v2 import OpenClawBridge

# Create bridge (uses config.json automatically)
bridge = OpenClawBridge()

# Create your first agent (any name)
bridge.create_agent("my_context")

# Save a conversation
bridge.save_conversation(
    agent="my_context",
    role="user",
    content="I need to remember this important insight"
)

# Recall context when needed
context = bridge.recall_context(
    agents=["my_context"],
    query="What important insights do I have?"
)

print(context)

Done! Your memory system is running. ✓

📦 What's Inside

Pre-Configured Settings

Everything is optimized from day 1:

{
  "session": {
    "max_tokens": 160000,        ← Session capacity
    "compact_trigger": 160000,   ← When to compress
    "compact_reserve": 40000     ← Keep after compression
  },
  "extraction": {
    "before_compact": true,      ← MANDATORY before compressing
    "auto_extract": true,        ← Automatic knowledge capture
    "save_to_vectors": true      ← Persist extracted knowledge
  }
}

Extraction Rules (Built-In)

Knowledge is automatically extracted using proven patterns:

Lessons learned:

"I learned that X is faster"
"Discovered that Y works better"
"Key insight: Z approach is more efficient"

Decisions made:

"Decided to use LanceDB instead"
"Chosen approach: vector-first architecture"
"We agreed to implement X pattern"

Projects & goals:

"Building a memory system"
"Working on improving performance"
"Project: system optimization"

Compaction Strategy

When session grows too large (160K tokens):

1. Extract all lessons/decisions/projects
2. Save to dedicated vector agents
3. Compact session (discard conversations, keep 40K reserve)
4. Continue with fresh token budget

Result: No knowledge loss, just cleaned-up conversation history.

🎯 Core Components

Memory Manager

from memory_manager_v2 import MemoryManager

manager = MemoryManager()  # Uses config.json
manager.create_agent("finance")
manager.add_memory_to_agent(
    agent="finance",
    content="Bitcoin reached new ATH"
)

results = manager.search_agent_memory(
    agent="finance",
    query="Bitcoin price history"
)

OpenClaw Bridge

from openclaw_bridge_v2 import OpenClawBridge

bridge = OpenClawBridge()

# Save conversation turns
bridge.save_conversation(agent="context", role="user", content="...")
bridge.save_conversation(agent="context", role="assistant", content="...")

# Recall relevant context
context = bridge.recall_context(
    agents=["context", "decisions"],
    query="What did we decide about X?"
)

Session Compactor

from session_compactor import SessionCompactor

compactor = SessionCompactor()

# Check if compaction needed
if compactor.should_compact(current_tokens=165000):
    # Automatically extracts lessons/decisions before compacting
    result = compactor.compact("my_agent", token_count=165000)
    print(f"Extracted {result['lessons_extracted']} lessons")

⚙️ Configuration Guide

config.json (Main Settings)

What it controls:

• Session size (how much memory per session)
• Compaction triggers (when to compress)
• Extraction settings (what to extract)
• Search defaults (query limits)
• Persistence (backups, logging)

Pre-configured values are production-ready. Only change if you have specific needs:

{
  "session": {
    "max_tokens": 160000,          // Increase for longer sessions
    "compact_trigger": 160000,     // Can be lower to compact earlier
    "compact_reserve": 40000       // Token budget after compaction
  },
  "extraction": {
    "before_compact": true,        // CRITICAL - don't disable
    "auto_extract": true,          // Automatic extraction
    "save_to_vectors": true        // Persist extracted knowledge
  },
  "search": {
    "default_limit": 5,            // Results per search
    "confidence_threshold": 0.7    // Min similarity score (0-1)
  }
}

rules/extraction-rules.json (What to Extract)

Customizable patterns for extracting knowledge:

{
  "lessons": {
    "keywords": ["learned", "discovered", "insight", "found"],
    "patterns": [
      "I learned that {content}",
      "Key takeaway: {content}",
      "Insight: {content}"
    ],
    "confidence": 0.9,
    "enabled": true
  },
  "decisions": {
    "keywords": ["decided", "chose", "agreed", "committed"],
    "patterns": [
      "Decided to {action}",
      "We chose {option}",
      "Agreement: {decision}"
    ],
    "confidence": 0.85,
    "enabled": true
  },
  "projects": {
    "keywords": ["building", "working on", "project", "developing"],
    "patterns": [
      "Building {name}",
      "Project: {description}",
      "Working on {item}"
    ],
    "confidence": 0.8,
    "enabled": true
  }
}

How to customize:

1. Add new keywords
2. Add new patterns (use {content} or {action} placeholders)
3. Adjust confidence levels (0.0-1.0)
4. Enable/disable categories

rules/compaction-rules.json (Compression Strategy)

Control how compaction happens:

{
  "strategy": "aggressive",      // "conservative", "balanced", "aggressive"
  "token_limits": {
    "max_session": 160000,       // Maximum before mandatory compaction
    "preserve_after": 40000,     // Token budget after compaction
    "warning_threshold": 140000  // Warn when approaching limit
  },
  "extraction": {
    "mandatory": true,           // CRITICAL - always extract before compacting
    "save_extracted_to_agents": true,
    "update_memory_md": true
  },
  "deletion": {
    "policy": "oldest_first",    // How to select what to delete
    "keep_recent_count": 50,     // Keep most recent N memories
    "preserve_high_confidence": true
  },
  "scheduling": {
    "auto_compact": true,        // Automatic trigger at max_session
    "compact_on_startup": false, // Check on system start
    "notify_before_compact": true
  }
}

rules/agent-templates.json (Agent Definitions)

Define agent templates users can create from:

{
  "conversation": {
    "description": "Store conversations and chat history",
    "type": "general",
    "extract_lessons": true,
    "extract_decisions": true,
    "retention_days": 30,
    "example": "bridge.create_agent('conversation')"
  },
  "decisions": {
    "description": "Critical decisions made",
    "type": "decisions",
    "extract_lessons": false,
    "extract_decisions": true,
    "retention_days": 90,
    "example": "bridge.create_agent('decisions')"
  },
  "lessons": {
    "description": "Lessons learned and insights",
    "type": "lessons",
    "extract_lessons": true,
    "extract_decisions": false,
    "retention_days": 365,
    "example": "bridge.create_agent('lessons')"
  },
  "temporary": {
    "description": "Temporary context (cleared on compaction)",
    "type": "temporary",
    "extract_lessons": false,
    "extract_decisions": false,
    "retention_days": 1,
    "example": "bridge.create_agent('temp_research')"
  }
}

📊 How It Works

Data Flow: Save & Recall

Your Code
    ↓
save_conversation(agent, role, content)
    ↓
Text → Auto-Embed (384-dim vector)
    ↓
Insert into LanceDB (agent's vector space)
    ↓
✓ Saved

---

recall_context(agents=["x", "y"], query="?")
    ↓
Query → Auto-Embed (same 384-dim space)
    ↓
Semantic search across agents
    ↓
Return top-5 similar memories + content
    ↓
Ready to use in LLM prompt

Compaction Flow: Mandatory Extraction

Session Token Count > 160K
    ↓
MANDATORY EXTRACTION PHASE
    ├─ Extract lessons using patterns
    ├─ Extract decisions using patterns
    ├─ Extract projects using patterns
    └─ Save to vector agents
    ↓
COMPACTION PHASE
    ├─ Delete old conversation memories
    ├─ Keep 40K token reserve
    └─ Preserve extracted vectors
    ↓
SESSION RESET
    ├─ Fresh 160K token budget
    ├─ Can still recall lessons/decisions from previous sessions
    └─ Ready to continue

Key: Extraction happens BEFORE deletion. No knowledge loss.

🔧 Customization Examples

Example 1: Change Token Limits

Want bigger/smaller memory?

// config.json
{
  "session": {
    "max_tokens": 200000,        // ← Bigger (more conversation history)
    "compact_trigger": 200000,
    "compact_reserve": 60000     // ← More preserved after compact
  }
}

Example 2: Add Custom Extraction Pattern

Want to extract something specific?

// rules/extraction-rules.json
{
  "bug_reports": {
    "keywords": ["bug", "error", "crash", "issue"],
    "patterns": [
      "Bug: {description}",
      "Error encountered: {details}"
    ],
    "confidence": 0.8,
    "enabled": true
  }
}

Then update memory_extractor.py:

def extract_bugs(self, text: str) -> List[str]:
    """Extract bug reports"""
    patterns = [
        r"Bug:\s*(.+?)(?:\.|,|$)",
        r"Error encountered:\s*(.+?)(?:\.|,|$)"
    ]
    # ... extraction logic

Example 3: Create Domain-Specific Agents

bridge = OpenClawBridge()

# Create agents for your domains
bridge.create_agent("trading_signals")
bridge.create_agent("market_analysis")
bridge.create_agent("risk_management")

# Save to specific agents
bridge.save_conversation(
    agent="trading_signals",
    role="system",
    content="BTC showing bullish divergence on 4h chart"
)

# Search across domain
context = bridge.recall_context(
    agents=["trading_signals", "market_analysis"],
    query="What are current trading signals?"
)

🚀 Production Deployment

Checklist

• Dependencies installed: pip install -r requirements.txt
• Setup run: python3 code/setup.py
• Tests pass: python3 code/test_full_system.py
• config.json reviewed (customize if needed)
• rules/ files reviewed (optional customization)
• Agents created: bridge.create_agent("name")
• Integration code written (save/recall in your agent)
• Monitoring setup (logs enabled, backups scheduled)

Logging & Monitoring

import logging

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    handlers=[
        logging.FileHandler('memory.log'),
        logging.StreamHandler()
    ]
)

# System will log:
# - Memory additions
# - Searches performed
# - Extractions triggered
# - Compactions executed
# - Performance metrics

Backup Strategy

Automatic backups configured in config.json:

{
  "persistence": {
    "auto_save": true,
    "backup_interval_hours": 24,
    "backup_path": "./backups",
    "compression": "gzip"
  }
}

Backups include:

• Vector database snapshot
• Configuration files
• Extracted knowledge (lessons/decisions)
• Metadata

📈 Performance

After model loads: Everything runs locally at 1-10ms.

❓ FAQ

Q: Can I change extraction rules? A: Yes! Edit rules/extraction-rules.json and restart.

Q: What if I need bigger session? A: Increase max_tokens in config.json.

Q: Does it work with OpenClaw? A: Yes, OpenClawBridge designed specifically for OpenClaw.

Q: Can I backup my memories? A: Automatic backups configured. Manual: code/backup.py.

Q: How much disk space? A: ~1.5GB per 1 million vectors. Scales linearly.

Q: Is it production-ready? A: Yes. Used in production. See deployment checklist above.

📚 File Structure

memory-complete/
├── SKILL.md                    ← You are here
├── README.md                   ← Quick overview
├── LICENSE                     ← MIT
├── requirements.txt            ← Dependencies
├── config.json                 ← MAIN CONFIG (pre-configured)
├── rules/
│   ├── extraction-rules.json   ← What to extract
│   ├── compaction-rules.json   ← How to compact
│   └── agent-templates.json    ← Agent definitions
├── code/
│   ├── embeddings.py           ← Hugging Face wrapper
│   ├── lance_memory.py         ← Vector DB
│   ├── memory_manager_v2.py    ← Orchestrator
│   ├── openclaw_bridge_v2.py   ← OpenClaw integration
│   ├── session_compactor.py    ← Compaction logic
│   ├── memory_extractor.py     ← Extraction engine
│   ├── rules_engine.py         ← Load & apply rules
│   ├── setup.py                ← Initialization
│   └── test_full_system.py     ← Tests
├── references/
│   ├── CONFIG_GUIDE.md         ← Detailed config options
│   ├── RULES_GUIDE.md          ← Custom rules tutorial
│   ├── IMPLEMENTATION_GUIDE.md ← Integration patterns
│   └── ARCHITECTURE.md         ← System design
└── memory/
    ├── vector_db/              ← LanceDB (created by setup)
    ├── backups/                ← Auto backups
    └── logs/                   ← System logs

🎓 Next Steps

1. Clone & Setup (3 minutes)

   git clone <repo>
   pip install -r requirements.txt
   python3 code/setup.py
   

2. Integrate (5-10 minutes)

   • Import OpenClawBridge
   • Create agents
   • Add save/recall calls

3. Customize (optional)

   • Edit config.json for your session sizes
   • Edit rules/extraction-rules.json for custom patterns
   • Add domain-specific agents

4. Deploy (production)

   • Enable logging
   • Configure backups
   • Monitor token usage
   • Iterate based on feedback

🤝 Support

• SKILL.md — This file (overview)
• README.md — Quick start
• references/CONFIG_GUIDE.md — All config options
• references/RULES_GUIDE.md — Customization tutorial
• GitHub Issues — Report problems

Clone. Setup. Use. Customize if needed.

That's the entire flow. Everything else is optional optimization.

Ready? → See README.md for quick start.