02 LLM Wiki Ingest

LLM Wiki — layer 02, ingest

You maintain a wiki that an LLM writes and an LLM reads. Layer 01 did every read and write in one context, which capped it at ten notes per run. This layer removes the cap two ways: page writing moves into subagents, so no single context ever holds every source, and an adapter layer lets anything with a URI become a source — local notes, web articles, GitHub repos.

Your job as orchestrator changes accordingly: you route inputs, spawn writers, read their receipts, and run the tail. You never read a raw file yourself.

Three files carry the contract, and you follow them exactly:

• CONVENTIONS.md — layout, identity, threshold, links, log, OKF alignment.
• PAGES.md — the template and receipt for every page type.
• SOURCES.md — the adapter contract, the routing table, and how to add an origin.

Read all three before your first write in a session.

Step 0 — Locate the wiki

Glob wiki-*/ in the project root and keep the ones that contain both raw/ and wiki/.

Creating a wiki:

mkdir -p wiki-<slug>/raw wiki-<slug>/wiki/sources wiki-<slug>/wiki/entities wiki-<slug>/wiki/concepts
printf '# Log\n' > wiki-<slug>/log.md

Step 1 — Detect mode

Ingest iff the user used an explicit verb (ingest, add, build, index) or dropped file/directory paths. Everything else is query.

When it is ambiguous, choose query. Redirecting a query is cheap; a mis-triggered ingest writes files the user did not ask for.

Spawning subagents

Every page in this layer is written by a subagent defined in agents/. To spawn one, whatever your harness calls it:

1. Read agents/<name>.md.
2. Pass its entire body as the subagent's prompt, followed by the inputs its Inputs table lists, as plain key: value lines.
3. Run it on a Sonnet-class model (or your provider's equivalent mid-tier model; fall back to the harness default). The agent file states its own preference.
4. Run them in parallel when your harness supports it, sequentially when it does not. The result is identical, only slower.
5. Expect exactly one JSON receipt back. If an agent returns prose instead, re-read the receipt shape in its file and re-run that one agent.

(In Claude Code: the Task/Agent tool, one call per writer, several in one message to run them concurrently.)

Why this is the whole lesson: a raw file is read exactly once, by one source_writer, and never enters your context or any other agent's. You see receipts — a few hundred tokens each. page_writer reads only source pages; overview_writer reads only frontmatter. That is why 50 sources cost about the same orchestrator context as 5.

Ingest path

You route and delegate. The only files you write yourself are log.md and the report to the user.

1.0 Route each input

Match every input against the routing table in SOURCES.md, first match wins:

The YouTube adapter is a deliberate skeleton: it raises NotImplementedError with a pointer to SOURCES.md § How to add a source. When a user drops a YouTube URL, run it, report the failure plainly, and continue with the other inputs.

1.1 Collect inputs and dedup

Expand every path the user gave you:

• a directory → recursively every *.md inside it. An assets/ folder is not a list of inputs — nothing inside it becomes a source. Its files travel later, in 1.2, as attachments of the notes that embed them.
• a file → itself.

For each input compute its raw path from the routing table — for a local note raw/<slug>.md, where <slug> is the filename stem, slugified: lowercase, every run of non-alphanumeric characters collapsed to a single -, leading/trailing - stripped, truncated to 60 characters at a - boundary. The adapters derive their own slugs the same way, from the URL's last path segment or from <owner>-<repo>.

Then split the inputs (this is the whole of dedup — see CONVENTIONS.md §4):

ls wiki-<slug>/raw/ wiki-<slug>/raw/repos/     # everything already ingested, by identity

• raw path exists → skipped. Report the input by name plus the original_path recorded on its existing page. If the incoming file differs in content from raw/, say so explicitly and still skip.
• raw path free → new.
• repo that already exists → refreshed, not skipped (CONVENTIONS.md §7).

There is no cap in this layer. Layer 01 capped a run at 10 notes because one context had to hold all of them; here each raw file is read by its own subagent and you only ever see receipts. Do sanity-check the count with the user before spawning 50 agents, and batch them in groups your harness can actually run.

1.2 Run the adapter for each new input

One adapter call per input, per the recipes in SOURCES.md:

# local — the note, then the attachments it embeds (see CONVENTIONS.md §4)
cp "<input path>" "wiki-<slug>/raw/<slug>.md"
mkdir -p "wiki-<slug>/raw/assets"
cp "<dir of the input note>/assets/<embedded file>" "wiki-<slug>/raw/assets/"   # skip if it exists

# article
uv run --script .agents/skills/02-llm-wiki-ingest/scripts/fetch_article.py \
  --url "<url>" --wiki-dir wiki-<slug>

# repo
uv run --script .agents/skills/02-llm-wiki-ingest/scripts/clone_repo.py \
  --repo "<url>" --wiki-dir wiki-<slug>

Keep each receipt — the writers need original_path, title, source_url, authors, published_date, and for repos clone_path, commit_sha, branch. If an article receipt carries a warning, surface it and do not write a source page from a paywall stub.

1.3 Spawn one writer per new raw artifact

List the existing pages once, and pass them to every writer so they reuse canonical slugs instead of inventing near-duplicates:

ls wiki-<slug>/wiki/entities wiki-<slug>/wiki/concepts

• File artifacts → one agents/source_writer.md per raw file, in parallel.
• Repos → one agents/repo_writer.md (mode: architecture) per repo, output wiki/repos/github-<owner>-<repo>/ARCHITECTURE.md.

Collect the receipts. Union of entities_referenced and concepts_referenced across them is your "touched slugs" set for the tail. Do not read the pages the writers produced — the receipt is the interface.

1.4 Ingest tail — entity and concept pages

uv run --script .agents/skills/02-llm-wiki-ingest/scripts/count_mentions.py --wiki-dir wiki-<slug>

The qualifying lists are the slugs at ≥2 distinct source-like pages — now counted across wiki/sources/ and wiki/repos/*/, so a repo can be one of the two sources that materialize a concept.

Spawn one agents/page_writer.md per qualifying slug that one of this run's new pages touched, in parallel, passing the source_pages the script listed for that slug. A slug that was already qualifying and that no new page mentions has not changed — leave it alone.

1.5 Rewrite the overview

Spawn agents/overview_writer.md with the count_mentions.py output. It walks frontmatter and reads at most 5 full pages.

1.6 Rebuild the index

uv run --script .agents/skills/02-llm-wiki-ingest/scripts/build_index_md.py --wiki-dir wiki-<slug>

Fix anything it reports as an ERROR: (a page with no type) before moving on.

1.7 Append to the log

## <YYYY-MM-DD> ingest | <what came in>

- Ingested N notes from `<path>`; skipped M already present (<names>)
- Source pages: <slugs>
- New pages: <slug> (concept, 2 sources), ...
- Updated pages: <slug> (2 → 3 sources), ...
- Waiting at 1 mention: <slug>, <slug>

1.8 Report back

New, skipped and refreshed counts by origin, the pages your agents wrote, the wiki path, and one line telling the user to open wiki-<slug>/wiki/index.md in Obsidian. If any slug is sitting at exactly 1 mention, name a couple — that is the wiki's own to-do list.

Query path

Read-only. The only file query mode writes in this layer is log.md.

• Q.1 — Locate the wiki (Step 0).
• Q.2 — Load the map. Read wiki/index.md, then the one or two wiki/<subdir>/index.md files that look relevant. Each bullet is a title — description; that is enough to choose.
• Q.3 — Read 1–3 pages. Prefer entity/concept pages (already synthesized) over source pages (single-note views). For "how does this code work" questions, start at wiki/repos/<repo>/ARCHITECTURE.md — it exists so nobody has to read the clone.
• Q.4 — Escalate only if you must. If the wiki pages genuinely do not answer the question, read the specific raw/ file a source page points to. Never bulk read raw/. For a repo page that means specific files under its clone_path; clones are never committed, so if it is missing, clone it yourself with clone_repo.py (same call as in 1.2) — never ask the user to. If the wiki simply does not know, say so — a wiki that hallucinates coverage is worse than an empty one.
• Q.5 — Answer with [[wikilinks]] to every page you used, and end with a Pages used: list. Distinguish what the notes claim from what you inferred.
• Q.6 — Log it:

## <YYYY-MM-DD> query | <topic>

- question: "<verbatim, ≤200 chars>"
- pages used: <count> (<slugs>)
- gap noticed: <one line, or omit>

Harness notes

This skill is written for any agent harness. It needs five capabilities:

In Claude Code those are Read, Write, Bash, AskUserQuestion and the Task/Agent tool. Nothing in this skill pins a model or a tool name — the agent files state a model preference in prose, and any harness that can run a sub-conversation with a prompt can run them.

Agent reference

Read the file, pass its body as the prompt, collect one JSON receipt.

Script reference

All five scripts are PEP 723 single files — uv installs their dependencies on the first run. You run them; the user never does. Paths below are relative to the layer directory.

No script writes wiki content. They fetch, they count, they render — every word in wiki/ is written by an agent.