# Book RAG Push > Phase 04b of the books pipeline — fan out the canonical `chunks.jsonl` to three vector stores (OpenAI Vector Stores, Gemini File Search, Anthropic Files API) and the in-house pgvector. Idempotent — only re-uploads chunks whose `sha256` changed since the last run; updates `vector_store_ids` map in chapter frontmatter to record which store/version each chapter lives in. Includes the smoke test (same query against all four stores, diff results). Use after `/book-chunk` produces an updated `chunks.jsonl`. Trigger phrases: "push to vector stores", "upload chunks", "phase 04 fan-out", "rag deploy", "smoke test rag". - Skill: `joshuashepherd/book-rag-push` (Agent Skill) - Install (CLI): `npx skillmds@latest add joshuashepherd/book-rag-push` - Raw SKILL.md: https://api.skillmd.com/api/skills/joshuashepherd/book-rag-push/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: JoshuaShepherd (https://skillmd.com/u/joshuashepherd) - Updated: 2026-09-08 - Page: https://skillmd.com/skills/joshuashepherd/book-rag-push --- # Book RAG Push — Phase 04b: Chunks → 3 Vector Stores ## Purpose Take the canonical `/.ingest/chunks.jsonl` produced by `/book-chunk` and fan it out to four destinations: 1. **pgvector** (`book_chunks` table in Supabase) — primary, with HNSW cosine index. 2. **OpenAI Vector Stores** — accessed via `file_search` tool in Responses API. 3. **Gemini File Search** — accessed via `fileSearch` tool in `generateContent`. 4. **Anthropic Files API** — accessed via `document` blocks with citations + memory tool for agentic sessions. Each chunk's `sha256` is the source of truth for "has this chunk changed". We only re-upload what's new or modified. ## Invocation `$ARGUMENTS`: - **``** — Required. - **`--targets `** — Comma-separated. Default: `pgvector,openai,gemini,claude`. - **`--dry-run`** — Print upload plan without making API calls. - **`--smoke-query `** — Final smoke-test query. Default: `"What is this book mostly about?"`. ## Process ### Step 1 — Pre-flight 1. Verify `chunks.jsonl` exists under `/.ingest/`. If not, run `/book-chunk` first. 2. Verify env vars per target: `OPENAI_API_KEY`, `OPENAI_VECTOR_STORE_ID` (if existing), `GEMINI_API_KEY`, `ANTHROPIC_API_KEY`, `DATABASE_URL` (for pgvector). 3. Read previous run's manifest (if any) at `/.ingest/rag-state.json`. Build a set of `id → sha256` for last-uploaded state per target. ### Step 2 — Fan out per target (parallel where safe) #### 2a. pgvector (in-house) ```typescript import { db } from "@/lib/database"; import { bookChunks } from "@/lib/database/schema/books"; for (const chunk of changedChunks) { const embedding = await embed(`${chunk.context_prefix}\n\n${chunk.text}`); await db.insert(bookChunks).values({ chapter_id: chapterIdMap[chunk.chapter_slug], section_id: sectionIdMap[chunk.section_anchor], ordinal: chunk.ordinal, content: chunk.text, context_prefix: chunk.context_prefix, page_start: chunk.page_start, page_end: chunk.page_end, token_count: chunk.token_count, embedding, embedding_model: "voyage-3-large", embedding_version: manifest.rag.embedding_version, content_sha256: chunk.sha256, }).onConflictDoUpdate({ target: [bookChunks.chapter_id, bookChunks.ordinal], set: { content: chunk.text, embedding, content_sha256: chunk.sha256 }, }); } ``` #### 2b. OpenAI Vector Stores One vector store per book per language: `book::`. Pre-chunk and pass each chunk as its own `.md` file with tight `static` chunking bounds so OpenAI's default chunker doesn't re-split across our boundaries. ```typescript const store = await openai.vectorStores.create({ name: `book:${manifest.slug}:${manifest.language}`, metadata: { book_slug: manifest.slug, language: manifest.language, version: manifest.version }, }); for (const batch of chunkBatches(changedChunks, 500)) { await openai.vectorStores.fileBatches.uploadAndPoll(store.id, { files: batch.map((c) => new File( [`${c.context_prefix}\n\n\n${c.text}`], `${c.id}.md`, { type: "text/markdown" } )), chunking_strategy: { type: "static", static: { max_chunk_size_tokens: 600, chunk_overlap_tokens: 100 } }, attributes: batch.map((c) => c.metadata) as any, }); } ``` Cap `max_num_results: 12` at query time (default 20 is too costly). #### 2c. Gemini File Search One File Search store per book: `book::`. Storage uses ~3× source size; query-time embeddings are free. Hard limit: 100MB per file, ≤20GB per store. For a multi-book corpus, one store per book per language stays well under. ```typescript const store = await ai.fileSearchStores.create({ displayName: `book:${manifest.slug}:${manifest.language}`, }); for (const c of changedChunks) { await ai.fileSearchStores.documents.upload(store.name, { file: `${c.context_prefix}\n\n\n${c.text}`, mimeType: "text/markdown", metadata: { book_slug: c.book_slug, chapter: `${c.chapter}`, page_start: `${c.page_start}` }, chunkingConfig: { maxTokensPerChunk: 600, overlapTokens: 80 }, }); } ``` #### 2d. Anthropic Files API Claude has no first-party vector store. Pattern: upload one PDF/MD file per **chapter** (not per chunk — too many) to Files API; rely on your own pgvector + Voyage rerank for retrieval; pass top-k `document` blocks with `citations.enabled: true`. Use the memory tool to persist retrieved context across an agentic session. ```typescript const file = await client.beta.files.upload({ file: await Bun.file(`/${slug}-ch03-missional-dna.md`).arrayBuffer(), purpose: "document", }); // store file.id in vector_store_ids.claude per chapter ``` ### Step 3 — Update vector_store_ids in chapter frontmatter After each successful upload, update each chapter's frontmatter: ```yaml vector_store_ids: openai: vs_abc123 gemini: fileSearchStores/fw-en-v1 claude: file_xyz789 last_indexed_at: 2026-04-28T18:30:00Z ``` Recompute `content_sha256` only if frontmatter content changed (it should, but `last_indexed_at` shouldn't trigger rechunking — see Issue 16 of the corresponding fix prompts). ### Step 4 — Persist run state Write `/.ingest/rag-state.json`: ```json { "ran_at": "2026-04-28T18:30:00Z", "targets": { "pgvector": { "uploaded": 12, "skipped": 4225, "store": "supabase://book_chunks" }, "openai": { "uploaded": 12, "skipped": 4225, "store": "vs_abc123" }, "gemini": { "uploaded": 12, "skipped": 4225, "store": "fileSearchStores/fw-en-v1" }, "claude": { "uploaded": 12, "skipped": 4225, "store": "files_xyz789" } }, "manifest_version": "1.2.0", "embedding_version": "v3" } ``` ### Step 5 — Smoke test Run the same query against all four stores; print results side by side. ```typescript const query = process.argv["--smoke-query"] ?? "What is this book mostly about?"; const pg = await ragSearchPgvector(query, { book_slug, k: 5 }); const oai = await openai.responses.create({ model: "gpt-5", tools: [{ type: "file_search", vector_store_ids: [openaiStoreId], max_num_results: 5 }], input: query, }); const gem = await ai.models.generateContent({ model: "gemini-2.5-pro", contents: query, tools: [{ fileSearch: { fileSearchStoreNames: [geminiStoreName] } }], }); const cla = await client.messages.create({ model: "claude-opus-4-7", max_tokens: 1024, messages: [ { role: "user", content: [ { type: "document", source: { type: "file", file_id: claudeFileId }, citations: { enabled: true } }, { type: "text", text: query }, ]}, ], }); console.table({ pgvector: pg.topIds, openai: oai.topCitations, gemini: gem.topCitations, claude: cla.topCitations }); ``` Verify: - All four stores return at least 1 citation. - The top-3 citations from each store have non-trivial Jaccard overlap (≥ 30%). - No store returns the same exact ID for unrelated queries (sanity check on attribute filtering). ### Step 6 — Report ``` Phase 04b — RAG fan-out pgvector: +12 chunks (4237 total in book_chunks) OpenAI: +12 files (vs_abc123 — 4237 files) Gemini: +12 docs (fileSearchStores/fw-en-v1 — 4237 docs) Claude: +0 files (chapter-level — no chapters changed) Smoke test: "What is this book mostly about?" pgvector: ch03-0007, ch01-0002, ch07-0014 ✓ OpenAI: ch03-0007, ch01-0002, ch04-0009 ✓ Gemini: ch03-0007, ch01-0002, ch07-0014 ✓ Claude: ch03-0007, ch04-0009, ch01-0002 ✓ Top-3 Jaccard: 0.62 (✓ ≥ 0.30) Verdict: PASS — book is live in all four stores ``` ## Versioning Bump `book.json:version` whenever: - Chapters added or removed. - Pagination changes. - Embedding model or chunking strategy changes (also bump `embedding_version`). The ingest script reads `chunks.jsonl` and only re-uploads chunks whose `sha256` changed — keeps store costs flat across copy edits. ## Cost Notes (April 2026) | Provider | Per-book cost (300pp) | Notes | |---|---|---| | pgvector + Voyage | ~$0.50 (embeddings) + storage | One-time embed; Voyage `voyage-3-large` ~$0.18/1M tokens | | OpenAI Vector Stores | ~$0.50 indexing + per-query embeddings | Pre-chunk + small static bounds keeps cost predictable | | Gemini File Search | ~$0.50 indexing + free queries | **Cheapest at query time** — query embeddings free, only model I/O charged | | Anthropic Files | $0 indexing + per-query input tokens | Pay nothing to upload; pay full input tokens per query | Gemini wins on multi-query economics; pgvector wins on full control + filtering. ## Out of Scope - Generating chunks — `/book-chunk`. - Schema validation — `/book-frontmatter`. - Stripe entitlement checks — separate concern in the e-reader app. ## References - [docs/html/books-pipeline.html#phase-4](../../../docs/html/books-pipeline.html#phase-4) - [OpenAI Vector Stores](https://developers.openai.com/api/reference/typescript/resources/vector_stores) - [Gemini File Search](https://ai.google.dev/gemini-api/docs/file-search) - [Claude Files API](https://platform.claude.com/docs/en/build-with-claude/files) - [Voyage rerank-2.5](https://docs.voyageai.com/docs/reranker)