Sanctuary Memory

Sanctuary Memory Configuration

Status: Active Domain: Project Sanctuary Depends on: memory-management (generic tiered pattern), rlm-factory, vector-db, obsidian-integration, huggingface-utils

Purpose

This skill maps the generic memory-management tiered architecture to Project Sanctuary's full storage stack. It knows every backend, file path, and plugin responsible for each memory tier.

The Complete Memory Stack

┌─────────────────────────────────────────────────────────┐
│  HOT CACHE (always in context at boot)                  │
│  Files: .agent/learning/*                               │
│  ~200 lines total                                       │
├─────────────────────────────────────────────────────────┤
│  SEMANTIC CACHE (fast lookup, loaded on demand)         │
│  Backend: rlm-factory → rlm_summary_cache.json          │
│  Backend: rlm-factory → rlm_tool_cache.json             │
├─────────────────────────────────────────────────────────┤
│  VECTOR STORE (semantic search + source code parsing)   │
│  Backend: vector-db → ChromaDB on port 8110             │
│  Profile: vector_profiles.json                          │
├─────────────────────────────────────────────────────────┤
│  DEEP STORAGE (filesystem, loaded on demand)            │
│  LEARNING/topics/, ADRs/, 01_PROTOCOLS/                 │
├─────────────────────────────────────────────────────────┤
│  VAULT (Obsidian, loaded on demand)                     │
│  Backend: obsidian-integration → OBSIDIAN_VAULT_PATH    │
│  Notes, canvases, graph connections                     │
├─────────────────────────────────────────────────────────┤
│  SOUL (external persistence, synced periodically)       │
│  Backend: huggingface-utils → HF Hub dataset            │
│  Repo: richfrem/Project_Sanctuary_Soul                  │
│  Structure: lineage/, data/, metadata/                  │
└─────────────────────────────────────────────────────────┘

Tier 1: Hot Cache (Boot Files)

Loaded in order at every session start:

Target: ~200 lines total across these 4 files.

Tier 2: Semantic Cache (RLM Factory)

Plugin: rlm-factory Config: .agent/learning/rlm_profiles.json

Refresh: /rlm-factory_gap-fill (Agent injection) OR python plugins/rlm-factory/skills/rlm-curator/scripts/distiller.py (Local Ollama batch)

Gap-Fill: Zero-Cost Bulk Strategy

When hundreds of files are uncached, use the Copilot swarm (free) rather than Ollama or paid Claude:

# Always use source ~/.zshrc -- NOT 'gh auth token' (lacks Copilot scope)
source ~/.zshrc
python3 plugins/agent-loops/skills/agent-swarm/scripts/swarm_run.py \
  --engine copilot \
  --job plugins/rlm-factory/resources/jobs/rlm_chronicle.job.md \
  --files-from rlm_distill_tasks_project.md \
  --resume --workers 2   # max 2 workers for Copilot rate limits

For higher throughput, use --engine gemini --workers 5 (also free tier).

Cache Safety: Concurrent Write Rule

inject_summary.py uses fcntl.flock (exclusive OS file lock) to serialize concurrent writes. Never run parallel cache writers without this lock -- two workers loading, then overwriting the same JSON will silently destroy each other's entries (race condition).

Cache Recovery: Checkpoint Reconciliation

If a batch run is interrupted and the cache is partially lost, reconcile before resuming:

# 1. Restore from best git snapshot:
git show <commit>:.agent/learning/rlm_summary_cache.json > /tmp/git_cache.json
# Merge: git as base, current as override
merged = {**git_cache, **current_cache}

# 2. Remove phantom checkpoint entries not in the real cache:
st['completed'] = [f for f in st['completed'] if f in merged.keys()]
st['failed'] = {}

# 3. Re-run with --resume -- it will re-process only the missing files

Tier 3: Vector Store (ChromaDB)

Plugin: vector-db Config: vector_profiles.json (pointing to vector_knowledge_manifest.json) Server: ChromaDB on localhost:8110 (Native Python Server)

The Vector DB provides Parent-Child semantic retrieval, returning full documents/files based on tiny 400-char conceptual embeddings. By combining scopes, it mirrors RLM coverage but extends it into deep code analysis using AST parsing.

Tier 4: Deep Storage (Filesystem)

Tier 5: Vault (Obsidian)

Plugin: obsidian-integration Config: OBSIDIAN_VAULT_PATH env var Guardian skill: sanctuary-obsidian-integration

Tier 6: Soul (HuggingFace)

Plugin: huggingface-utils Guardian skill: sanctuary-soul-persistence Dataset: richfrem/Project_Sanctuary_Soul

Tags: project-sanctuary, cognitive-continuity, reasoning-traces, ai-memory, llm-training-data, metacognition

Memory Flow by Session Phase

Boot (Phase I)

1. Load hot cache (Tier 1)
2. Iron check validates snapshot integrity
3. If stale → flag for refresh

During Session

• New learning → Tier 4 (LEARNING/topics/)
• Need context → Tier 2 (RLM query) → Tier 3 (vector search) → Tier 4 (file read)
• New decision → adr-manager → Tier 4
• New protocol → protocol-manager → Tier 4
• Journal entry → chronicle-manager → Tier 4

Closure (Phase VI-IX)

1. Seal → Update snapshot (Tier 1), capture state
2. Persist → Upload to HuggingFace (Tier 6)
3. Ingest → Refresh RLM cache (Tier 2) + Vector DB (Tier 3)
4. Vault export → Optionally write to Obsidian (Tier 5)