Ingesting Resources

Ingesting Resources

Promotes a source from {corpus-root}/sources/ingest/ (staging) to {corpus-root}/sources/original/ (audited) by running the 9-pass protocol against it.

Each successful invocation produces, all inside the active corpus:

• One sidecar at sources/original/{slug}.source.md with frontmatter (title, author, URL, scope, licence, ingested, original_format, checksum_sha256).
• Converted markdown at sources/converted/{slug}.md (the "tape" Pass C read).
• One deep reference at references/{slug}-deep.md.
• One light reference at references/{slug}.md.
• One distillation per applicable task axis at distillations/{task}/{slug}-{task}.md. The protocol forms an opinion about applicability and asks the operator to confirm borderline cases.
• Updates to the runtime JSON indexes (references/slug-table.json, reference-index.json, concept-index.json, per-task distillations/{task}/task-index.json) and the operator-inspection .md views alongside.
• Substantive images extracted and indexed in {corpus-root}/sources/converted/IMAGE-INDEX.yaml.
• An ingestion summary.
• The input source removed from sources/ingest/. Directory state is the workflow indicator: anything in ingest/ is unaudited; anything in original/ has been through the protocol.

{corpus-root} is operator-supplied (e.g., corpus.commons/demo/). Default: active corpus from working-directory inference, or builds.yaml's defaults.source_dir.

Per-pass procedures, templates, and failure modes are in 9-pass-protocol.md.

When to use

• Adding a new source to the corpus.
• Re-ingesting after a substantive revision (new edition, corrected text, previous Pass I failure).
• Re-ingesting when a model advance makes the deep ref materially better.

When not to use

• Editing an existing reference's prose. Substantive edits rerun Pass C onwards; cosmetic edits rerun Pass I at minimum.
• Adding a new task axis without a new source. Run creating-tasks → creating-applications instead.
• Searching the corpus for an existing source. Run matching-references instead.

Cost and time

The protocol pays its full cost upfront so query-time is selection, not synthesis. Before you run it, know what you are spending. Numbers below are from the May 2026 OpenStax pilot (12 books, 165–995 source pages each, mean ~480 pages); detail at docs/architecture/ingestion-protocol.md.

• Per source, single ingestion: ~28 min wall-clock when run alongside other work; ~$1–5 in Opus 4.7 tokens at xhigh effort for a book-length source. A short source (a paper, a chapter) lands at the low end. A long source with many task axes lands at the high end.
• Per corpus, parallel ingestion (4–5 subagents): ~5h 30min wall-clock and an Opus token bill in the low hundreds of dollars for a 12-source corpus. Roughly doubles with corpus size; scales with axis count at Pass G.
• Re-ingestion (revised edition, model advance, prior Pass I failure) costs as much as the first run. Plan re-ingestions deliberately. Do not rerun for cosmetic reasons.

The cost lives in generation, not in re-reading the source. Within one ingestion session, Pass C reads the source once. Passes D–I work from the deep reference. Collapsing the passes into one mega-prompt does not save tokens (the same tokens are generated either way) and it costs the audit surface: Pass I checks against one inspectable artefact per pass. The amortisation runs across queries, not across passes. A source ingested once is read by however many queries the corpus serves over its life.

If the budget is tight, ingest fewer sources at higher quality. The matrix's value is in curation, not coverage. The architectural answer to is this source worth ingesting? sits upstream of this skill — see finding-resources.

Setup gates

Confirm before any pass runs.

Accept input

File path mode. A path to a PDF, EPUB, or pre-extracted markdown file in {corpus-root}/sources/ingest/.

/ingesting-resources corpus.local/my-corpus/sources/ingest/some-book.pdf
/ingesting-resources corpus.commons/demo/sources/ingest/new-source.epub

Paste mode. Literal text pasted with explicit source metadata. The pasted text is written to {corpus-root}/sources/ingest/{slug}.md first, then promoted by the protocol.

/ingesting-resources paste --corpus corpus.local/my-corpus

The operator provides: pasted source text, plus author, title, edition, year, ISBN or DOI.

Flags.

• --corpus {path}: set the active corpus root. Default: infer from input path, or builds.yaml's defaults.source_dir. All output artefacts land inside this corpus root.
• --full / --text-only: pre-commit the image-scope decision (pre-flight check 4). When absent, prompt the operator.

/ingesting-resources corpus.local/my-corpus/sources/ingest/some-book.pdf --text-only
/ingesting-resources paste --corpus corpus.local/my-corpus --full

Required metadata (either mode):

• Author (primary; secondary authors as a list).
• Title and subtitle.
• Edition / year of publication.
• ISBN or DOI where available.
• Source format and condition (PDF / EPUB / extracted markdown; conversion notes if any).
• Source licence (exact string: CC BY 4.0, CC BY-NC-SA 4.0, All rights reserved, etc., or unverified).
• Distribution scope: open, open-nc, copyrighted, confidential, or personal. Usually correlates with licence (CC BY → open, CC BY-NC-SA → open-nc, all-rights-reserved or unverified → copyrighted) but can diverge. The build excludes references whose scope exceeds a profile's max_scope; personal ships in no profile. Both fields land on the deep ref at Pass A: licence in **Source:**, scope on **Scope:**.

Source preparation. Pre-extract to markdown for better citation-anchor preservation. Converted markdown lands in {corpus-root}/sources/converted/.

• PDF: default converter markitdown (pip install 'markitdown[all]'); fallback when markitdown fails: pymupdf4llm.
• EPUB: do NOT use markitdown. On EPUB, markitdown silently extracts only the navigation/heading layer and drops the flowing body text — observed dropping ~93–95% of the body across several EPUBs (2026-07-14) while leaving the chapter headings intact, so the output looks complete. Use calibre instead: ebook-convert input.epub output.txt --txt-output-formatting=markdown, then rename .txt → .md (calibre keys output format off the file extension and rejects .md directly). Record converted_via: calibre-ebook-convert.

Mandatory post-conversion sanity check (before any pass reads the file). Silent-truncation converters are the highest-severity source-integrity failure because the loss is invisible downstream. After converting, run wc -w on the output and compare against the source's expected length (a ~250-page book is ~65,000–90,000 words; a ~100-page book ~35,000–40,000; roughly 250–400 words/page). If the word count is a small fraction of the expected length (e.g. a full book converting to a few thousand words), the converter dropped body text — stop, re-convert with a different tool, and re-check. Never hand a suspiciously-short conversion to an ingestion pass or a subagent. This check is what turns the completeness pre-flight (below) from a heading-structure glance into a coverage guarantee.

Pre-invocation: check for duplicate

Before invoking, run matching-references for the source's central topic. If a reference for this source already exists in the corpus, do not re-ingest unless one of the "When to use" conditions applies (substantive revision, model advance, prior Pass I failure).

Source-integrity pre-flight (mandatory)

The first three checks below stop the run if they fail; the fourth is an operator decision recorded in the deep ref.

1. Confirm completeness — hard stop on incomplete sources. Verify the source is whole without committing to a full read yet. Check: the table of contents matches actual section headers, chapter-boundary markers are present, no obvious gaps or truncation appear (a final chapter cut mid-paragraph is a clear signal). Enumerate the full chapter list from the TOC and record it in the deep ref's source/structure block — this anchor list becomes the audit target at Pass I.

If anything is missing or partial, stop the run and escalate to the operator. Report the specific gap (which chapters are missing, why — scanner truncation, publisher's partial edition, conversion bug). The operator decides whether to (a) get a complete source and restart, (b) explicitly approve partial-coverage ingestion with a written rationale, or (c) abandon the ingestion.

The coverage: frontmatter field is operator-consent only. Absence means complete. Presence requires coverage: partial — operator-approved YYYY-MM-DD — reason with the operator's reason recorded inline. The model never sets coverage: partial unilaterally. A reader-side decision to skip readable chapters is not a "partial coverage" outcome; it is a source-integrity violation. If you can read a chapter, you must read it; if you cannot finish the read in the available context budget, stop and escalate so the operator decides whether to continue, defer, or abandon.

2. Detect citation style. Grep for page anchors ({page markers in pandoc output, or numeric page boundaries in the converted markdown):

• Page anchors preserved: cite as (Ch N, p. M).
• Page anchors lost in conversion: cite as (Ch N, "Section name").

Per-source decision. Record it in the deep reference's source/structure block.

3. Confirm model identity. Opus 4.7 or higher. Record the operator's session model in the deep ref's frontmatter (Pass A). If the protocol fans out to subagents, each subagent declares its model identity in its first response; on any mismatch, stop and report.

4. Image scope decision. Declare up front whether this run includes image classification:

• Full. Every extracted image is visually inspected and either added to IMAGE-INDEX.yaml (substantive) or deleted (decorative).
• Text-only. Skip image classification this run. Record in the deep reference's coverage line. Complete later via ingesting-images.

No default. If neither --full nor --text-only was passed, prompt the operator. Never start full and truncate mid-classification on context budget: that produces silent partial coverage and violates source-integrity. If full looks too large for one session, declare text-only and run ingesting-images later.

Effort and quality policy

Every downstream artefact (light, distillation, indexes) derives from the deep. Pass I is the gate; nothing ships before Pass I passes on the deep.

The 9 passes

Run in order. Full procedure, templates, contracts, and pass-specific failure modes are in 9-pass-protocol.md.

• Pass A, Context. Metadata, structure, citation-style block, frontmatter.
• Pass B, Structural read. Read once at speed; build an internal map.
• Pass C, Deep read with citations. Read the source in full; attach citations to every non-trivial claim.
• Pass D, Blockquote extraction. Select short verbatim passages; verify character-by-character.
• Pass E, Synthesis. Assemble the deep reference; apply [V] [AP] [AR] [AE] [BT] markers.
• Pass F, Light-reference derivation. Three sub-passes from the verified deep; source not re-read.
• Pass G, Distillation projection. Applicability gate (G.0); project per applicable axis (G.1–G.3).
• Pass H, Cross-reference. Drive the mechanical-index pipeline: allocate slug-ID, run preprocessor, dispatch Sonnet refs + cross-link passes, regenerate JSON indexes. Cross-check sibling distillations.
• Pass I, Source-only audit. Read the deep cold in a fresh subagent — never the session that produced it — and trace every claim. A same-context read certifies its own leakage: a session cannot see the training-priors it just wrote in as source-grounded, so it passes them off as verified. Only independent eyes catch them. Deep ships only when Pass I passes.

Parallel-batch operations

When ingesting many sources in one work-session, dispatch multiple subagents in parallel: one source per subagent for Passes A–H. Pass I always runs in a separate fresh subagent (per Pass I above), launched once the producing subagent has written the deep reference — a cold read is only cold from a context that did not write the reference.

Per-source staging files. Each source agent writes its per-source artefacts to namespaced staging paths the build scripts read at corpus level. The canonical JSON indexes are derived; nobody edits them by hand. Staging layout (paths relative to repo root):

• _planning/extracted/{corpus}/{slug}.json: deterministic preprocessor output (one per source).
• _planning/staging/{corpus}/refs/{slug}.json: Sonnet refs-pass output (one per source).
• _planning/staging/{corpus}/concepts/candidates.json, decisions.json: corpus-wide concept aggregation (single file, written once after every per-source step completes).
• docs/images/_ingest_image_index_{slug}.yaml: image-index staging (one per source). Assembled into {corpus-root}/sources/converted/IMAGE-INDEX.yaml by scripts/build_indexes/build_image_index.py at Pass H; an image absent from that index is not part of the library, so the assembly step is not optional. This staging path sits at repo root rather than inside the corpus, which means a private corpus's per-figure descriptions land in a shared tree — docs/images/ is gitignored to keep them off any public remote.

The build scripts (scripts/build_indexes/) consume the staging artefacts and write the canonical JSON indexes in corpus.commons/{corpus}/. Two reasons for the staging layer:

• No race on the JSON indexes. Per-source extractions and Sonnet refs passes run in parallel; the build scripts merge at corpus level after every agent completes.
• Intermediate work survives partial failure. If a source agent's session times out or rate-limits before its final report, staging artefacts remain on disk; the operator resumes without re-running the preprocessor.

Concurrency ceiling. Around 4-5 simultaneous Opus 4.7 ingestion subagents. Beyond that, Anthropic-side rate limits drop requests. Wrap-up subagents (Pass H + I on already-text-complete sources) tolerate slightly more.

A stalled agent is usually not a failed agent — check disk before re-running. Stall notifications fire on stream silence, not on death. Across a 2026 batch run, 5 of 7 agents reported failed had completed their writes; separately, two agents that had died left the deep reference edited and stamped with no audit log. Both mistakes are expensive in opposite directions, and both are settled the same way: look at the files. Compare content hashes and modification times against what the agent was asked to produce, and re-run only what is genuinely absent.

Where an agent dies matters more than that it died. A half-finished ingestion leaves no broken grammar — it leaves confidently wrong classifications under an unearned stamp, and the retry inherits them with no way to tell what it is looking at. Have agents write their log or index artefact before reporting, and stamp only after that artefact exists, so a stamp always implies a completed pass.

A different model family is worth having in the pool, for two separate reasons. It is the only thing that catches Pass I's prior-leakage blind spot (see Pass I in 9-pass-protocol.md). It also runs outside the Anthropic rate limit, so it is genuine additional capacity rather than a share of the same budget — five concurrent sessions ran without interference in 2026 testing — and it proved markedly more robust on very large tapes, completing 48k- and 72k-word source reads that had defeated several same-family agents. Expect its client to time out while it keeps working server-side: check disk before retrying, because the files usually landed.

Post-pass operations

After Pass I passes on the deep and Passes F, G, H have produced the derived artefacts, four operations finalise the ingestion.

Promote source from ingest/ to original/

Directory state is the workflow indicator: anything in sources/ingest/ is unaudited; anything in sources/original/ has been through the protocol.

1. Author the .source.md sidecar at {corpus-root}/sources/original/{slug}.source.md with frontmatter populated from Pass A metadata:

   ---
   title: {from Pass A}
   author: {from Pass A — primary author, et al. for multi-author works}
   publisher: {from Pass A, if known}
   publish_year: {YYYY}
   isbn: {if known}
   url: {canonical URL or DOI}
   scope: {open | open-nc | copyrighted | confidential | personal}
   licence: {exact licence string from Pass A}
   ingested: {YYYY-MM-DD of this run}
   original_format: {pdf | epub | docx | md | ...}
   converted_via: {markitdown <version> | pymupdf4llm <version> | manual | ...}
   checksum_sha256: {sha256 of the input file, if binary}
   ---
   
   # Source: {Title} ({Author})
   
   {2–4 line body: provenance prose, where the source was acquired,
   any conversion caveats, links to converted markdown and deep ref.}
   

2. Write the converted markdown to {corpus-root}/sources/converted/{slug}.md (if not already present from a manual pre-conversion). This is the "tape" Pass C read; Pass C's deep reference traces against this file.

3. Do NOT automatically copy the binary from ingest/ to original/. Whether to ship the original is a redistribution decision depending on licence and operator intent. The build's .gitignore ignores corpus.commons/*/sources/original/*.{pdf,epub,mobi,azw3}; operators with rights to redistribute can git add -f after this step.

4. Remove the input file from {corpus-root}/sources/ingest/. The sidecar and converted markdown carry the audited record forward.

Failure recovery. If any pass fails, leave the input file in ingest/ and remove any partial output artefacts from original/, converted/, references/, distillations/. The ingest/ directory state is the recovery signal: anything still there is unfinished business.

Image extraction

Runs only if pre-flight scope is full. Skip if text-only; ingesting-images handles it later.

Ordering: image extraction must run before the input file is removed from ingest/.

1. Run python3 scripts/extract-images.py {corpus-root}/sources/ingest/{source}.pdf before removing the input.
2. The extractor writes images to {corpus-root}/sources/converted/{source-slug}-images/ and an extraction manifest to {corpus-root}/sources/converted/extraction-manifest.json.
3. Classify each extracted image by visual inspection (Read tool on the image file):
   • SUBSTANTIVE: diagram, chart, table, model, framework, process flow, decision tree, formula illustration, or other figure carrying conceptual content. Add an entry to {corpus-root}/sources/converted/IMAGE-INDEX.yaml with description, tags, suggested-use, and framework cross-references.
   • DECORATIVE: photographs, cover art, ornamental rules, blank frames, publisher logos, paper textures, icons. Delete from {corpus-root}/sources/converted/{source-slug}-images/.
4. Image files are gitignored. The index travels with the repo. If an image is not in IMAGE-INDEX.yaml, it is not part of the library.

Per-image visual inspection is mandatory. Never bulk-classify by filename, file extension, file size, or any metadata-only signal: the same format carries diagrams in one source and photographs in another.

Build verification

npm run build

Both profiles should produce non-empty {corpus-root}/apps/{profile}/ and pass referential integrity. The build's sources pair-check validates that every sources/converted/{slug}.md has a sources/original/{slug}.source.md sibling; a fresh ingestion should land both atomically.

Ingestion summary

Output a one-screen summary covering:

• Source ingested (citation).
• Pass-by-pass results (durations, claim counts, audit fail count before fixes).
• Files produced: .source.md sidecar path, converted markdown path, deep ref path, light ref path, distillation file paths.
• Applicability decisions per task axis: applied, skipped (with reason), operator-confirmed on borderline calls.
• Indexes updated.
• Images extracted, classified SUBSTANTIVE, classified DECORATIVE.
• Promotion confirmed: input file removed from sources/ingest/ (or, on partial failure, input remains in ingest/ for re-run; name this explicitly).
• Open follow-ups (e.g., partial-coverage note in frontmatter flagged for re-ingestion when missing chapters arrive).

Related skills

• creating-corpus: scaffold a new corpus before ingesting into it.
• finding-resources: triage candidates against the protocol's preconditions before dropping into sources/ingest/.
• matching-references: search the corpus for an existing source before ingesting a duplicate.
• creating-distillations: per-distillation projection; invoked by this skill's Pass G.
• creating-applications: assemble a new application (build profile + distillation index + orchestrated per-distillation distillation).
• creating-tasks: scope a new task axis. Upstream of creating-applications.