Book RAG Push

Book RAG Push — Phase 04b: Chunks → 3 Vector Stores

Purpose

Take the canonical <book-dir>/.ingest/chunks.jsonl produced by /book-chunk and fan it out to four destinations:

1. pgvector (book_chunks table in Supabase) — primary, with HNSW cosine index.
2. OpenAI Vector Stores — accessed via file_search tool in Responses API.
3. Gemini File Search — accessed via fileSearch tool in generateContent.
4. Anthropic Files API — accessed via document blocks with citations + memory tool for agentic sessions.

Each chunk's sha256 is the source of truth for "has this chunk changed". We only re-upload what's new or modified.

Invocation

$ARGUMENTS:

• <book-slug-or-dir> — Required.
• --targets <list> — Comma-separated. Default: pgvector,openai,gemini,claude.
• --dry-run — Print upload plan without making API calls.
• --smoke-query <text> — Final smoke-test query. Default: "What is this book mostly about?".

Process

Step 1 — Pre-flight

1. Verify chunks.jsonl exists under <book-dir>/.ingest/. If not, run /book-chunk first.
2. Verify env vars per target: OPENAI_API_KEY, OPENAI_VECTOR_STORE_ID (if existing), GEMINI_API_KEY, ANTHROPIC_API_KEY, DATABASE_URL (for pgvector).
3. Read previous run's manifest (if any) at <book-dir>/.ingest/rag-state.json. Build a set of id → sha256 for last-uploaded state per target.

Step 2 — Fan out per target (parallel where safe)

2a. pgvector (in-house)

import { db } from "@/lib/database";
import { bookChunks } from "@/lib/database/schema/books";

for (const chunk of changedChunks) {
  const embedding = await embed(`${chunk.context_prefix}\n\n${chunk.text}`);
  await db.insert(bookChunks).values({
    chapter_id: chapterIdMap[chunk.chapter_slug],
    section_id: sectionIdMap[chunk.section_anchor],
    ordinal: chunk.ordinal,
    content: chunk.text,
    context_prefix: chunk.context_prefix,
    page_start: chunk.page_start,
    page_end: chunk.page_end,
    token_count: chunk.token_count,
    embedding,
    embedding_model: "voyage-3-large",
    embedding_version: manifest.rag.embedding_version,
    content_sha256: chunk.sha256,
  }).onConflictDoUpdate({
    target: [bookChunks.chapter_id, bookChunks.ordinal],
    set: { content: chunk.text, embedding, content_sha256: chunk.sha256 },
  });
}

2b. OpenAI Vector Stores

One vector store per book per language: book:<slug>:<lang>. Pre-chunk and pass each chunk as its own .md file with tight static chunking bounds so OpenAI's default chunker doesn't re-split across our boundaries.

const store = await openai.vectorStores.create({
  name: `book:${manifest.slug}:${manifest.language}`,
  metadata: { book_slug: manifest.slug, language: manifest.language, version: manifest.version },
});

for (const batch of chunkBatches(changedChunks, 500)) {
  await openai.vectorStores.fileBatches.uploadAndPoll(store.id, {
    files: batch.map((c) => new File(
      [`${c.context_prefix}\n\n<!-- page: ${c.page_start} -->\n${c.text}`],
      `${c.id}.md`,
      { type: "text/markdown" }
    )),
    chunking_strategy: { type: "static", static: { max_chunk_size_tokens: 600, chunk_overlap_tokens: 100 } },
    attributes: batch.map((c) => c.metadata) as any,
  });
}

Cap max_num_results: 12 at query time (default 20 is too costly).

2c. Gemini File Search

One File Search store per book: book:<slug>:<lang>. Storage uses ~3× source size; query-time embeddings are free. Hard limit: 100MB per file, ≤20GB per store. For a multi-book corpus, one store per book per language stays well under.

const store = await ai.fileSearchStores.create({
  displayName: `book:${manifest.slug}:${manifest.language}`,
});
for (const c of changedChunks) {
  await ai.fileSearchStores.documents.upload(store.name, {
    file: `${c.context_prefix}\n\n<!-- page: ${c.page_start} -->\n${c.text}`,
    mimeType: "text/markdown",
    metadata: { book_slug: c.book_slug, chapter: `${c.chapter}`, page_start: `${c.page_start}` },
    chunkingConfig: { maxTokensPerChunk: 600, overlapTokens: 80 },
  });
}

2d. Anthropic Files API

Claude has no first-party vector store. Pattern: upload one PDF/MD file per chapter (not per chunk — too many) to Files API; rely on your own pgvector + Voyage rerank for retrieval; pass top-k document blocks with citations.enabled: true. Use the memory tool to persist retrieved context across an agentic session.

const file = await client.beta.files.upload({
  file: await Bun.file(`<book-dir>/${slug}-ch03-missional-dna.md`).arrayBuffer(),
  purpose: "document",
});
// store file.id in vector_store_ids.claude per chapter

Step 3 — Update vector_store_ids in chapter frontmatter

After each successful upload, update each chapter's frontmatter:

vector_store_ids:
  openai: vs_abc123
  gemini: fileSearchStores/fw-en-v1
  claude: file_xyz789
last_indexed_at: 2026-04-28T18:30:00Z

Recompute content_sha256 only if frontmatter content changed (it should, but last_indexed_at shouldn't trigger rechunking — see Issue 16 of the corresponding fix prompts).

Step 4 — Persist run state

Write <book-dir>/.ingest/rag-state.json:

{
  "ran_at": "2026-04-28T18:30:00Z",
  "targets": {
    "pgvector": { "uploaded": 12, "skipped": 4225, "store": "supabase://book_chunks" },
    "openai":   { "uploaded": 12, "skipped": 4225, "store": "vs_abc123" },
    "gemini":   { "uploaded": 12, "skipped": 4225, "store": "fileSearchStores/fw-en-v1" },
    "claude":   { "uploaded": 12, "skipped": 4225, "store": "files_xyz789" }
  },
  "manifest_version": "1.2.0",
  "embedding_version": "v3"
}

Step 5 — Smoke test

Run the same query against all four stores; print results side by side.

const query = process.argv["--smoke-query"] ?? "What is this book mostly about?";

const pg = await ragSearchPgvector(query, { book_slug, k: 5 });
const oai = await openai.responses.create({
  model: "gpt-5",
  tools: [{ type: "file_search", vector_store_ids: [openaiStoreId], max_num_results: 5 }],
  input: query,
});
const gem = await ai.models.generateContent({
  model: "gemini-2.5-pro",
  contents: query,
  tools: [{ fileSearch: { fileSearchStoreNames: [geminiStoreName] } }],
});
const cla = await client.messages.create({
  model: "claude-opus-4-7",
  max_tokens: 1024,
  messages: [
    { role: "user", content: [
      { type: "document", source: { type: "file", file_id: claudeFileId }, citations: { enabled: true } },
      { type: "text", text: query },
    ]},
  ],
});

console.table({ pgvector: pg.topIds, openai: oai.topCitations, gemini: gem.topCitations, claude: cla.topCitations });

Verify:

• All four stores return at least 1 citation.
• The top-3 citations from each store have non-trivial Jaccard overlap (≥ 30%).
• No store returns the same exact ID for unrelated queries (sanity check on attribute filtering).

Step 6 — Report

Phase 04b — RAG fan-out
  pgvector:  +12 chunks   (4237 total in book_chunks)
  OpenAI:    +12 files    (vs_abc123 — 4237 files)
  Gemini:    +12 docs     (fileSearchStores/fw-en-v1 — 4237 docs)
  Claude:    +0 files     (chapter-level — no chapters changed)

  Smoke test: "What is this book mostly about?"
    pgvector: ch03-0007, ch01-0002, ch07-0014   ✓
    OpenAI:   ch03-0007, ch01-0002, ch04-0009   ✓
    Gemini:   ch03-0007, ch01-0002, ch07-0014   ✓
    Claude:   ch03-0007, ch04-0009, ch01-0002   ✓
    Top-3 Jaccard: 0.62 (✓ ≥ 0.30)

  Verdict: PASS — book is live in all four stores

Versioning

Bump book.json:version whenever:

• Chapters added or removed.
• Pagination changes.
• Embedding model or chunking strategy changes (also bump embedding_version).

The ingest script reads chunks.jsonl and only re-uploads chunks whose sha256 changed — keeps store costs flat across copy edits.

Cost Notes (April 2026)

Gemini wins on multi-query economics; pgvector wins on full control + filtering.

Out of Scope

• Generating chunks — /book-chunk.
• Schema validation — /book-frontmatter.
• Stripe entitlement checks — separate concern in the e-reader app.

References

• docs/html/books-pipeline.html#phase-4
• OpenAI Vector Stores
• Gemini File Search
• Claude Files API
• Voyage rerank-2.5