Summarized RAG Memories

Summarized RAG Memory Ingestion Protocol (Atomic Fact Distillation)

Overview

When agents query external documentation or past conversation memory via Retrieval-Augmented Generation (RAG), standard vector retrieval returns top-K raw document chunks (typically 5 chunks $\times$ 1,000 tokens = 5,000 tokens).

Directly stuffing raw RAG chunks into the prompt causes severe issues:

1. Severe Token Waste: 80% of each 1,000-token chunk consists of generic background descriptions, boilerplate headers, and unrelated sections.
2. Context Dilution: The critical 1-sentence answer is buried inside 5,000 tokens of surrounding text, degrading frontier reasoning accuracy.
3. High Input Token Cost: Paid on every single turn where the memory context is injected.

The Summarized RAG Memory Protocol introduces a Fact Distillation Layer (via fast Tier-1 models or local token extractors) to condense raw retrieved chunks into 3 to 5 atomic key facts (80 to 120 tokens) before context ingestion.

Raw Vector Chunk Stuffing vs. Atomic Fact Distillation

┌─────────────────────────────────────────────────────────────┐
│                 RAG Ingestion Token Footprint               │
│                                                             │
│  Raw Vector Chunk Stuffing (5 Chunks / 5,200 Tokens):       │
│  • Chunk 1: AWS S3 SDK setup, credentials, imports (1,100t) │
│  • Chunk 2: Bucket ACL policies, XML examples      (1,050t) │
│  • Chunk 3: Presigned URL method signature          (950t)  │
│  • Chunk 4: Multipart upload troubleshooting       (1,100t) │
│  • Chunk 5: Deprecated v2 SDK migration notes      (1,000t) │
│  ↳ 5,200 tokens billed, high distraction noise              │
│                                                             │
│  Atomic Fact Distillation (3 Bullets / 85 Tokens - 98.4% Cut):│
│  • S3 Presigned URL API: `s3Client.getSignedUrlPromise(...)`│
│  • Expiration parameter: `Expires: 3600` (seconds)          │
│  • Required IAM Action: `s3:GetObject` on target ARN        │
│  ↳ 85 clean tokens, 100% relevant actionable signal         │
└─────────────────────────────────────────────────────────────┘

The 3-Step Fact Distillation Architecture

┌───────────────────────────────────────────────────────────────────────────┐
│ 1. RETRIEVE CANDIDATES: Dense vector / BM25 search fetches Top-5 chunks   │
│ 2. EXTRACT ATOMIC FACTS: Fast Micro-Tier (Haiku / Flash / Local GGUF)      │
│    distills chunks into strictly 3 high-density bullet points             │
│ 3. INJECT LEAN MEMORY: Primary Frontier Agent receives only the 3 bullets │
└───────────────────────────────────────────────────────────────────────────┘

Production Python Fact Distiller

from openai import OpenAI
from typing import List

client = OpenAI()

def distill_rag_chunks_to_facts(query: str, raw_chunks: List[str]) -> str:
    """Condenses 5,000 tokens of raw RAG chunks into 3 atomic bullet points."""
    combined_text = "\n\n---\n\n".join(raw_chunks)
    
    # Use Fast Tier 1 Model (GPT-4o-mini / Claude Haiku) for zero-latency distillation
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": (
                    "Extract strictly the top 3-4 atomic facts from the documentation "
                    "that directly answer the user query. Output strictly 3-4 bullet points. "
                    "No preamble, no conversational text."
                )
            },
            {
                "role": "user",
                "content": f"Query: {query}\n\nDocumentation Chunks:\n{combined_text}"
            }
        ],
        max_tokens=200,
        temperature=0.0
    )
    
    return response.choices[0].message.content.strip()

Master Ingestion Prompt Format

When feeding distilled facts to the primary coding agent:

<retrieved_memory>
- Database connection pool maximum size is set to 20 connections in `src/db/pool.ts`.
- Redis eviction policy is configured as `volatile-lru` with 1GB memory limit.
- Auth tokens expire after 900 seconds (15 minutes).
</retrieved_memory>

Benchmark Comparison

Processing 100 RAG-assisted technical queries:

Agent Operational Directive

MANDATORY: RAG systems must never feed raw multi-paragraph document chunks directly to frontier models. Always distill retrieved text into 3 to 5 atomic bullet points using a fast extraction tier before injecting into active agent context.