Book Chunk

Book Chunk — Phase 04a: Canonical chunks.jsonl

Purpose

Generate the single source of truth for retrieval — <book-dir>/.ingest/chunks.jsonl. This file feeds pgvector, OpenAI Vector Stores, Gemini File Search, and Claude Files (via /book-rag-push). One JSONL line per chunk; the only thing that varies per provider is the upload format.

The chunker uses the April-2026 SOTA combination:

• Structure-first — split on H2 / H3 headings; never break a paragraph, footnote, code block, or blockquote mid-content.
• Recursive target — 512 tokens per chunk, 64-token overlap (~12.5%).
• Late chunking (Jina, 2024) — embed the full chapter once with a long-context model, then mean-pool token vectors per chunk. Preserves cross-paragraph anaphora (proven +10–12% on documents with pronoun chains).
• Contextual prefix (Anthropic, Sept 2024) — prepend a 50–100 token LLM-generated context that situates the chunk in book + chapter; embed the prefixed text. Cuts retrieval failure ~67% per Anthropic's own benchmarks.

Invocation

$ARGUMENTS:

• <book-slug-or-dir> — Required.
• --target-tokens <N> — Default: 512. From book.json:rag.chunk_target_tokens.
• --overlap-tokens <N> — Default: 64. From book.json:rag.chunk_overlap_tokens.
• --changed-only — Only re-chunk chapters whose content_sha256 differs from the last chunks.jsonl entry. Default: true.
• --out <path> — Default: <book-dir>/.ingest/chunks.jsonl.

Process

Step 1 — Read inputs

1. Load <book-slug>-book.json → manifest.
2. Walk every <book-slug>-ch*.md file → frontmatter + body.
3. Verify each frontmatter parses against ChapterFrontmatter Zod schema; abort if not.

Step 2 — Structure split

For each chapter:

1. Parse with remark + unist-util-visit.
2. Walk the AST collecting groups: each group starts at an h2 or h3 heading and contains the heading + all content nodes until the next h2/h3.
3. Track heading_path per group (e.g. ["Missional DNA", "The Latent Code"]).
4. Capture page-marker <!-- page: N --> boundaries; each chunk records page_start (first marker in group) and page_end (last marker before next group).

Step 3 — Recursive split within groups

For groups exceeding target-tokens:

1. Split on paragraph boundaries first (blank lines).
2. If a paragraph alone exceeds target, split on sentence boundaries (NLP sentence tokenizer; never break inside a footnote, blockquote, or code block).
3. Apply overlap-tokens by repeating the last N tokens of chunk K at the start of chunk K+1.

Use tiktoken (cl100k_base) for token counting.

Step 4 — Generate contextual prefix per chunk

Use a small Claude model (e.g. claude-haiku-4-5) for cost. Prompt:

You are generating a 50-100 token context to prepend to a passage from a book,
to improve retrieval. Output ONLY the context (no preamble, no quotes).

Book: <manifest.title>
Chapter: <chapter_title>
Section path: <heading_path joined with " > ">
Page range: <page_start>-<page_end>

Passage:
<chunk text>

Context (50-100 tokens):

Cache the chapter-level summary to avoid regenerating shared context across chunks of the same chapter.

Step 5 — Late chunking embedding (optional, recommended)

Generate embeddings for pgvector ingestion:

1. Embed the full chapter (truncated to 8K tokens if needed) with voyage-3-large (or text-embedding-3-large if Voyage unavailable).
2. For each chunk, mean-pool the token vectors that fall within the chunk's character range.
3. Store the per-chunk embedding alongside the chunk.

If the embedding model does not return token-level vectors (most don't), fall back to per-chunk embedding of context_prefix + "\n\n" + text. This is what most providers actually use; "late chunking" with mean-pooling is only available for specific Jina / Voyage long-context endpoints.

Step 6 — Emit chunks.jsonl

One line per chunk:

{"id":"<book-slug>-ch03-0007","book_slug":"<book-slug>","chapter":3,"chapter_slug":"missional-dna","section_anchor":"#latent-code","page_start":79,"page_end":80,"heading_path":["Missional DNA","The Latent Code"],"text":"…the six elements of mDNA …","context_prefix":"From Hirsch's *The Forgotten Ways*, chapter 3 (\"Missional DNA\"), section \"The Latent Code\". This passage introduces the six elements of mDNA as the recoverable code beneath every Jesus movement.","token_count":498,"sha256":"a31b…","metadata":{"language":"en","key_concepts":["mdna","apostolic-genius"],"is_preview":false}}

Field reference:

• id — <book-slug>-ch<NN>-<NNNN> (zero-padded chapter + chunk ordinal). Stable across re-runs.
• book_slug, chapter, chapter_slug, section_anchor, page_start, page_end, heading_path — addressable provenance.
• text — the chunk body.
• context_prefix — Anthropic-style situating context.
• token_count — cl100k tokens.
• sha256 — hash of context_prefix + "\n\n" + text. Used by /book-rag-push to detect what's changed since last upload.
• metadata — language, key_concepts, is_preview, mentions_figures, mentions_scriptures.

Step 7 — Diff against previous run

If <book-dir>/.ingest/chunks.jsonl already exists:

1. Read the previous chunks; map id → sha256.
2. For each new chunk, compare. Mark __diff: "added" | "modified" | "unchanged" in a side-file .ingest/chunks.diff.json (do not modify the canonical jsonl with diff metadata).
3. Print a summary: +12 added, ~7 modified, 4218 unchanged.

Step 8 — Report

Phase 04a — Chunking
  Chapters chunked:    18 / 18
  Total chunks:        4237 (+12 added, ~7 modified, 4218 unchanged)
  Tokens / chunk:      avg 487, p95 511, max 528 (target 512)
  Overlap:             62 tokens avg (target 64)
  Context prefix gen:  $0.84 in API spend
  Output:              corpus/alan_hirsch/<book-slug>/.ingest/chunks.jsonl

  Verdict: PASS — advance to /book-rag-push

Quality Checks

• Every chunk's text is non-empty and under 600 tokens.
• Every chunk has a non-empty context_prefix between 50 and 150 tokens.
• No chunk crosses an h2 boundary (i.e., heading_path[0] is consistent within a chunk).
• No chunk breaks mid-footnote (verify by checking [^N] opens have matching closes within the same chunk).
• Every id is unique book-wide.
• Sum of chunk token_count ≥ 0.95 × sum of source chapter word_count × 1.3 (rough word→token ratio).

Out of Scope

• Pushing to providers — /book-rag-push.
• Inserting into book_chunks table — the next phase.

References

• docs/html/books-pipeline.html#phase-4
• Late chunking paper
• Anthropic Contextual Retrieval
• PreMAI 2026 chunking benchmark