Book Frontmatter — Phase 02: Markdown → MDX (frontmatter-validated)
Purpose
Bring every chapter's frontmatter into compliance with the corpus schema:
- Identity:
file_id (ULID), canonical_title, book_slug, chapter_number, chapter_slug, chapter_title, display_title.
- Integrity:
content_class, source_type, language, word_count, char_count, estimated_reading_time, page_range_start/end, content_sha256.
- RAG metadata:
key_concepts, themes, mentions_scriptures, mentions_figures, primary_topic, secondary_topics.
- Citations:
citations[] with id, kind (scripture | book | article), ref.
- Vector store linkage:
embedding_model, embedding_version, last_indexed_at, vector_store_ids map.
- Author + book identity:
author, co_authors, year, edition, publisher, isbn, tenant, tenant_id.
Validate against scripts/lib/frontmatter-schema.ts (Zod). Validate book.json against BookManifest. Block advancement to Phase 03 on any schema failure.
Invocation
$ARGUMENTS:
<book-slug-or-dir> — Required.
--mode <repair|validate-only|regenerate> — Default: repair.
repair — fix what's broken; preserve hand-edited fields.
validate-only — report violations without writing.
regenerate — rewrite all derived fields from scratch (preserves summary/[PENDING-REVIEW]).
--report <path> — Default: <book-dir>/.ingest/frontmatter-<YYYY-MM-DD>.json.
Process
Step 0 — Verify per-chapter granularity (CRITICAL)
Before generating frontmatter, confirm Phase 01 produced one file per book chapter. If a chapter file's body contains multiple ### Chapter N: markers, Phase 01 mis-split a part-hierarchy book and lumped multiple chapters into one file (see book-convert → "Books with PART hierarchy"). Fix this before frontmatter — generating frontmatter on a merged file produces wrong chapter_number, sparse coverage in the e-reader's chapter list, and chapter-level RAG queries that miss most of the book.
python3 -c "
import re, glob
problems = []
for f in glob.glob('corpus/alan_hirsch/<book-slug>/*.md'):
text = open(f).read()
# Skip frontmatter
if text.startswith('---\n'):
end = text.find('\n---\n', 4)
text = text[end+5:] if end > 0 else text
chapters = re.findall(r'^### Chapter \d+', text, re.MULTILINE)
if len(chapters) > 1:
problems.append((f, chapters))
for f, chs in problems:
print(f'MERGED: {f} contains {len(chs)} chapter markers: {chs}')
print('OK' if not problems else 'RUN PHASE 01 RE-SPLIT FIRST')
"
If anything prints MERGED:, run the part-hierarchy splitter (see book-convert Step 4) before continuing.
Step 1 — Inventory + parse
for f in corpus/alan_hirsch/<book-slug>/*.md; do
head -100 "$f" | python3 -c "
import sys, yaml
text = sys.stdin.read()
parts = text.split('---', 2)
if len(parts) >= 3:
fm = yaml.safe_load(parts[1])
print(f'{sys.argv[1]}: keys={sorted(fm.keys())}')
"
done
Step 2 — Recompute derived fields
For each chapter file (Python script):
import hashlib, re, yaml, sys
from pathlib import Path
def split_fm(text):
if not text.startswith('---\n'): return None, text
end = text.find('\n---\n', 4)
return text[:end+5], text[end+5:]
for path in Path('corpus/alan_hirsch/<book-slug>').glob('*.md'):
raw = path.read_text()
fm_block, body = split_fm(raw)
if not fm_block: continue
body_stripped = body.strip()
sha = hashlib.sha256(body_stripped.encode()).hexdigest()
chars = len(body_stripped)
words = len(body_stripped.split())
minutes = max(1, round(words / 200))
excerpt = ' '.join(
l for l in body.split('\n')
if l.strip() and not l.lstrip().startswith(('#', '>'))
)[:500]
new_fm = re.sub(r'(content_sha256:\s*).*', f'\\1{sha}', fm_block)
new_fm = re.sub(r'(word_count:\s*).*', f'\\1{words}', new_fm)
new_fm = re.sub(r'(char_count:\s*).*', f'\\1{chars}', new_fm)
new_fm = re.sub(r'(estimated_reading_time:\s*).*', f'\\1{minutes}', new_fm)
path.write_text(new_fm + body)
print(f'{path.name}: words={words}, sha={sha[:12]}…')
Step 3 — Validate against Zod
pnpm tsx scripts/validate-corpus.ts corpus/alan_hirsch/<book-slug> --schema-only
If the script does not exist or does not yet support --schema-only, write a quick Node.js wrapper that:
- Reads every
.md file's frontmatter
- Parses YAML
- Runs
ChapterFrontmatter.parse(fm) from scripts/lib/frontmatter-schema.ts
- Reports violations per file
For book.json, run BookManifest.parse(JSON.parse(text)).
Step 4 — Repair common defects
Frequent issues observed across the corpus:
| Defect |
Fix |
canonical_title contains kerning artifact (Mc Neal) |
Replace with correct form (McNeal) |
Opening excerpt reflects pre-edit body (e.g. fabricated paragraph) |
Regenerate from current body |
mentions_scriptures empty when body has scripture references |
Run scripture-detection regex (\b\d?\s?[A-Z][a-z]+\.?\s+\d+:\d+(-\d+)?\b) and populate |
mentions_figures missing well-known authors quoted in body |
Cross-reference body against a known-authors list |
chapter_number: 0 for non-front-matter chapter |
Renumber based on file's position in book |
summary: "[PENDING-REVIEW]…" |
Leave as-is — Phase 5 task per _prompts/00-CONTEXT.md |
content_sha256 doesn't match current body |
Recompute (Step 2) |
co_authors missing when book has multiple authors |
Add from book.json:author[] |
Step 5 — Update book.json totals
total_words = sum chapter.word_count
total_chars = sum chapter.char_count
update book.json: { total_chapters, total_word_count, total_char_count, last_validated_at }
Step 6 — Report
Phase 02 — Frontmatter
Files validated: 18 / 18
Schema failures: 0
Recomputed sha: 18
Recomputed word_count: 18
book.json: valid against BookManifest
Opening excerpts: 18 refreshed
PENDING-REVIEW summaries: 18 (preserved — Phase 5 task)
Verdict: PASS — advance to /book-ingest
Quality Checks
- Every
.md file has a parseable YAML frontmatter block.
- Every
content_sha256 matches sha256(body.strip()).
- Every
word_count matches len(body.strip().split()) ± 0.
- Every
chapter_number is a non-negative integer.
book.json is valid against BookManifest.
- No
Footnote reference \d+ placeholder strings (cf. Reframation Appendix 3).
- No
Mc Neal, Mome ntu m, or other kerning artifacts in canonical_title / display_title.
Out of Scope
- Body content fixes — handled by
/book-fix.
- Generating
summary Phase 5 prose — out of scope per project convention.
- ULID minting for new files — handled by
/book-convert at conversion time.
References
1---2name: book-frontmatter3description: >4 Phase 02 of the books pipeline — generate, repair, or recompute YAML5 frontmatter for every chapter in a book and validate against the Zod schema6 at `scripts/lib/frontmatter-schema.ts`. Recomputes `content_sha256`,7 `word_count`, `char_count`, `estimated_reading_time`, and `Opening excerpt`8 to match the current body. Validates `book.json` against `BookManifest` and9 every chapter against `ChapterFrontmatter`. Use after `/book-convert` +10 `/book-validate` pass, before `/book-ingest`. Trigger phrases: "fix the11 frontmatter", "regenerate frontmatter", "validate book.json", "phase 02",12 "schema-validate the corpus".13allowed-tools: Read, Write, Edit, Bash, Grep, Glob14metadata:15 source: https://skillmd.com/skills/joshuashepherd/book-frontmatter16 pipeline_phase: "02"17 reference: "docs/html/books-pipeline.html#phase-2"18---1920# Book Frontmatter — Phase 02: Markdown → MDX (frontmatter-validated)2122## Purpose2324Bring every chapter's frontmatter into compliance with the corpus schema:2526- **Identity:** `file_id` (ULID), `canonical_title`, `book_slug`, `chapter_number`, `chapter_slug`, `chapter_title`, `display_title`.27- **Integrity:** `content_class`, `source_type`, `language`, `word_count`, `char_count`, `estimated_reading_time`, `page_range_start/end`, `content_sha256`.28- **RAG metadata:** `key_concepts`, `themes`, `mentions_scriptures`, `mentions_figures`, `primary_topic`, `secondary_topics`.29- **Citations:** `citations[]` with `id`, `kind` (scripture | book | article), `ref`.30- **Vector store linkage:** `embedding_model`, `embedding_version`, `last_indexed_at`, `vector_store_ids` map.31- **Author + book identity:** `author`, `co_authors`, `year`, `edition`, `publisher`, `isbn`, `tenant`, `tenant_id`.3233Validate against `scripts/lib/frontmatter-schema.ts` (Zod). Validate `book.json` against `BookManifest`. Block advancement to Phase 03 on any schema failure.3435## Invocation3637`$ARGUMENTS`:3839- **`<book-slug-or-dir>`** — Required.40- **`--mode <repair|validate-only|regenerate>`** — Default: `repair`.41 - `repair` — fix what's broken; preserve hand-edited fields.42 - `validate-only` — report violations without writing.43 - `regenerate` — rewrite all derived fields from scratch (preserves `summary`/`[PENDING-REVIEW]`).44- **`--report <path>`** — Default: `<book-dir>/.ingest/frontmatter-<YYYY-MM-DD>.json`.4546## Process4748### Step 0 — Verify per-chapter granularity (CRITICAL)4950Before generating frontmatter, confirm Phase 01 produced one file per book chapter. If a chapter file's body contains *multiple* `### Chapter N:` markers, Phase 01 mis-split a part-hierarchy book and lumped multiple chapters into one file (see [book-convert](../book-convert/SKILL.md) → "Books with PART hierarchy"). **Fix this before frontmatter** — generating frontmatter on a merged file produces wrong `chapter_number`, sparse coverage in the e-reader's chapter list, and chapter-level RAG queries that miss most of the book.5152```bash53python3 -c "54import re, glob55problems = []56for f in glob.glob('corpus/alan_hirsch/<book-slug>/*.md'):57 text = open(f).read()58 # Skip frontmatter59 if text.startswith('---\n'):60 end = text.find('\n---\n', 4)61 text = text[end+5:] if end > 0 else text62 chapters = re.findall(r'^### Chapter \d+', text, re.MULTILINE)63 if len(chapters) > 1:64 problems.append((f, chapters))65for f, chs in problems:66 print(f'MERGED: {f} contains {len(chs)} chapter markers: {chs}')67print('OK' if not problems else 'RUN PHASE 01 RE-SPLIT FIRST')68"69```7071If anything prints `MERGED:`, run the part-hierarchy splitter (see [book-convert](../book-convert/SKILL.md) Step 4) before continuing.7273### Step 1 — Inventory + parse7475```bash76for f in corpus/alan_hirsch/<book-slug>/*.md; do77 head -100 "$f" | python3 -c "78import sys, yaml79text = sys.stdin.read()80parts = text.split('---', 2)81if len(parts) >= 3:82 fm = yaml.safe_load(parts[1])83 print(f'{sys.argv[1]}: keys={sorted(fm.keys())}')84"85done86```8788### Step 2 — Recompute derived fields8990For each chapter file (Python script):9192```python93import hashlib, re, yaml, sys94from pathlib import Path9596def split_fm(text):97 if not text.startswith('---\n'): return None, text98 end = text.find('\n---\n', 4)99 return text[:end+5], text[end+5:]100101for path in Path('corpus/alan_hirsch/<book-slug>').glob('*.md'):102 raw = path.read_text()103 fm_block, body = split_fm(raw)104 if not fm_block: continue105 body_stripped = body.strip()106 sha = hashlib.sha256(body_stripped.encode()).hexdigest()107 chars = len(body_stripped)108 words = len(body_stripped.split())109 minutes = max(1, round(words / 200))110 excerpt = ' '.join(111 l for l in body.split('\n')112 if l.strip() and not l.lstrip().startswith(('#', '>'))113 )[:500]114 new_fm = re.sub(r'(content_sha256:\s*).*', f'\\1{sha}', fm_block)115 new_fm = re.sub(r'(word_count:\s*).*', f'\\1{words}', new_fm)116 new_fm = re.sub(r'(char_count:\s*).*', f'\\1{chars}', new_fm)117 new_fm = re.sub(r'(estimated_reading_time:\s*).*', f'\\1{minutes}', new_fm)118 path.write_text(new_fm + body)119 print(f'{path.name}: words={words}, sha={sha[:12]}…')120```121122### Step 3 — Validate against Zod123124```bash125pnpm tsx scripts/validate-corpus.ts corpus/alan_hirsch/<book-slug> --schema-only126```127128If the script does not exist or does not yet support `--schema-only`, write a quick Node.js wrapper that:1291301. Reads every `.md` file's frontmatter1312. Parses YAML1323. Runs `ChapterFrontmatter.parse(fm)` from `scripts/lib/frontmatter-schema.ts`1334. Reports violations per file134135For `book.json`, run `BookManifest.parse(JSON.parse(text))`.136137### Step 4 — Repair common defects138139Frequent issues observed across the corpus:140141| Defect | Fix |142|---|---|143| `canonical_title` contains kerning artifact (`Mc Neal`) | Replace with correct form (`McNeal`) |144| `Opening excerpt` reflects pre-edit body (e.g. fabricated paragraph) | Regenerate from current body |145| `mentions_scriptures` empty when body has scripture references | Run scripture-detection regex (`\b\d?\s?[A-Z][a-z]+\.?\s+\d+:\d+(-\d+)?\b`) and populate |146| `mentions_figures` missing well-known authors quoted in body | Cross-reference body against a known-authors list |147| `chapter_number: 0` for non-front-matter chapter | Renumber based on file's position in book |148| `summary: "[PENDING-REVIEW]…"` | Leave as-is — Phase 5 task per `_prompts/00-CONTEXT.md` |149| `content_sha256` doesn't match current body | Recompute (Step 2) |150| `co_authors` missing when book has multiple authors | Add from `book.json:author[]` |151152### Step 5 — Update book.json totals153154```python155total_words = sum chapter.word_count156total_chars = sum chapter.char_count157update book.json: { total_chapters, total_word_count, total_char_count, last_validated_at }158```159160### Step 6 — Report161162```163Phase 02 — Frontmatter164 Files validated: 18 / 18165 Schema failures: 0166 Recomputed sha: 18167 Recomputed word_count: 18168 book.json: valid against BookManifest169 Opening excerpts: 18 refreshed170 PENDING-REVIEW summaries: 18 (preserved — Phase 5 task)171172 Verdict: PASS — advance to /book-ingest173```174175## Quality Checks176177- Every `.md` file has a parseable YAML frontmatter block.178- Every `content_sha256` matches `sha256(body.strip())`.179- Every `word_count` matches `len(body.strip().split())` ± 0.180- Every `chapter_number` is a non-negative integer.181- `book.json` is valid against `BookManifest`.182- No `Footnote reference \d+` placeholder strings (cf. Reframation Appendix 3).183- No `Mc Neal`, `Mome ntu m`, or other kerning artifacts in `canonical_title` / `display_title`.184185## Out of Scope186187- Body content fixes — handled by `/book-fix`.188- Generating `summary` Phase 5 prose — out of scope per project convention.189- ULID minting for new files — handled by `/book-convert` at conversion time.190191## References192193- [docs/html/books-pipeline.html#phase-2](../../../docs/html/books-pipeline.html#phase-2)194- [scripts/lib/frontmatter-schema.ts](../../../scripts/lib/frontmatter-schema.ts) — Zod schema source of truth195- [scripts/validate-corpus.ts](../../../scripts/validate-corpus.ts) — CLI validator