# 02 LLM Wiki Ingest > Build and query an LLM-maintained wiki from notes, articles and GitHub repos, with one subagent per source (modes: ingest, query). - Skill: `decodingai-magazine/02-llm-wiki-ingest` (Agent Skill, multi-file: 13 files) - Install (CLI): `npx skillmds@latest add decodingai-magazine/02-llm-wiki-ingest` - Raw SKILL.md: https://api.skillmd.com/api/skills/decodingai-magazine/02-llm-wiki-ingest/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: decodingai-magazine (https://skillmd.com/u/decodingai-magazine) - Updated: 2026-09-22 - Page: https://skillmd.com/skills/decodingai-magazine/02-llm-wiki-ingest --- # LLM Wiki — layer 02, ingest You maintain a wiki that an LLM writes and an LLM reads. Layer 01 did every read and write in one context, which capped it at ten notes per run. This layer removes the cap two ways: **page writing moves into subagents**, so no single context ever holds every source, and **an adapter layer** lets anything with a URI become a source — local notes, web articles, GitHub repos. Your job as orchestrator changes accordingly: you route inputs, spawn writers, read their receipts, and run the tail. **You never read a raw file yourself.** Three files carry the contract, and you follow them exactly: - `CONVENTIONS.md` — layout, identity, threshold, links, log, OKF alignment. - `PAGES.md` — the template and receipt for every page type. - `SOURCES.md` — the adapter contract, the routing table, and how to add an origin. Read all three before your first write in a session. --- ## Step 0 — Locate the wiki Glob `wiki-*/` in the project root and keep the ones that contain both `raw/` and `wiki/`. | Found | Do | |---|---| | exactly 1 | Use it. | | more than 1 | Ask the user which one. | | 0, ingest mode | Propose a slug from the topic of the incoming notes (kebab-case, e.g. `ai-engineering`), confirm it with the user, then create the wiki. | | 0, query mode | Say there is no wiki yet and stop. Do not create one to answer a question. | Creating a wiki: ```bash mkdir -p wiki-/raw wiki-/wiki/sources wiki-/wiki/entities wiki-/wiki/concepts printf '# Log\n' > wiki-/log.md ``` ## Step 1 — Detect mode **Ingest** iff the user used an explicit verb (*ingest, add, build, index*) **or** dropped file/directory paths. Everything else is **query**. When it is ambiguous, choose **query**. Redirecting a query is cheap; a mis-triggered ingest writes files the user did not ask for. --- ## Spawning subagents Every page in this layer is written by a subagent defined in `agents/`. To spawn one, whatever your harness calls it: 1. Read `agents/.md`. 2. Pass its **entire body as the subagent's prompt**, followed by the inputs its Inputs table lists, as plain `key: value` lines. 3. Run it on a Sonnet-class model (or your provider's equivalent mid-tier model; fall back to the harness default). The agent file states its own preference. 4. Run them **in parallel** when your harness supports it, sequentially when it does not. The result is identical, only slower. 5. Expect **exactly one JSON receipt** back. If an agent returns prose instead, re-read the receipt shape in its file and re-run that one agent. *(In Claude Code: the Task/Agent tool, one call per writer, several in one message to run them concurrently.)* Why this is the whole lesson: a raw file is read exactly **once**, by one `source_writer`, and never enters your context or any other agent's. You see receipts — a few hundred tokens each. `page_writer` reads only source pages; `overview_writer` reads only frontmatter. That is why 50 sources cost about the same orchestrator context as 5. --- # Ingest path You route and delegate. The only files you write yourself are `log.md` and the report to the user. ### 1.0 Route each input Match every input against the routing table in `SOURCES.md`, first match wins: | Input | Origin | Raw artifact | Writer | |---|---|---|---| | an existing path on disk | `local` | `raw/.md` | `source_writer` | | `github.com//` or `*.git` | `repo` | `raw/repos/.github--/` | `repo_writer` | | `youtube.com` / `youtu.be` | `youtube` | `raw/youtube-.md` | `source_writer` | | any other `http(s)://` | `article` | `raw/article-.md` | `source_writer` | The YouTube adapter is a deliberate skeleton: it raises `NotImplementedError` with a pointer to `SOURCES.md § How to add a source`. When a user drops a YouTube URL, run it, report the failure plainly, and continue with the other inputs. ### 1.1 Collect inputs and dedup Expand every path the user gave you: - a directory → **recursively every `*.md` inside it**. An `assets/` folder is not a list of inputs — nothing inside it becomes a source. Its files travel later, in 1.2, as attachments of the notes that embed them. - a file → itself. For each input compute its raw path from the routing table — for a local note `raw/.md`, where `` is the **filename stem**, slugified: lowercase, every run of non-alphanumeric characters collapsed to a single `-`, leading/trailing `-` stripped, truncated to 60 characters at a `-` boundary. The adapters derive their own slugs the same way, from the URL's last path segment or from `-`. Then split the inputs (this is the whole of dedup — see `CONVENTIONS.md` §4): ```bash ls wiki-/raw/ wiki-/raw/repos/ # everything already ingested, by identity ``` - raw path exists → **skipped**. Report the input by name plus the `original_path` recorded on its existing page. If the incoming file differs in content from `raw/`, say so explicitly and still skip. - raw path free → **new**. - **repo that already exists → refreshed, not skipped** (`CONVENTIONS.md` §7). There is **no cap in this layer.** Layer 01 capped a run at 10 notes because one context had to hold all of them; here each raw file is read by its own subagent and you only ever see receipts. Do sanity-check the count with the user before spawning 50 agents, and batch them in groups your harness can actually run. ### 1.2 Run the adapter for each new input One adapter call per input, per the recipes in `SOURCES.md`: ```bash # local — the note, then the attachments it embeds (see CONVENTIONS.md §4) cp "" "wiki-/raw/.md" mkdir -p "wiki-/raw/assets" cp "/assets/" "wiki-/raw/assets/" # skip if it exists # article uv run --script .agents/skills/02-llm-wiki-ingest/scripts/fetch_article.py \ --url "" --wiki-dir wiki- # repo uv run --script .agents/skills/02-llm-wiki-ingest/scripts/clone_repo.py \ --repo "" --wiki-dir wiki- ``` Keep each receipt — the writers need `original_path`, `title`, `source_url`, `authors`, `published_date`, and for repos `clone_path`, `commit_sha`, `branch`. If an article receipt carries a `warning`, surface it and do not write a source page from a paywall stub. ### 1.3 Spawn one writer per new raw artifact List the existing pages once, and pass them to every writer so they reuse canonical slugs instead of inventing near-duplicates: ```bash ls wiki-/wiki/entities wiki-/wiki/concepts ``` - **File artifacts** → one `agents/source_writer.md` per raw file, in parallel. - **Repos** → one `agents/repo_writer.md` (`mode: architecture`) per repo, output `wiki/repos/github--/ARCHITECTURE.md`. Collect the receipts. Union of `entities_referenced` and `concepts_referenced` across them is your "touched slugs" set for the tail. Do not read the pages the writers produced — the receipt is the interface. ### 1.4 Ingest tail — entity and concept pages ```bash uv run --script .agents/skills/02-llm-wiki-ingest/scripts/count_mentions.py --wiki-dir wiki- ``` The `qualifying` lists are the slugs at ≥2 distinct source-like pages — now counted across `wiki/sources/` **and** `wiki/repos/*/`, so a repo can be one of the two sources that materialize a concept. Spawn one `agents/page_writer.md` per qualifying slug **that one of this run's new pages touched**, in parallel, passing the `source_pages` the script listed for that slug. A slug that was already qualifying and that no new page mentions has not changed — leave it alone. ### 1.5 Rewrite the overview Spawn `agents/overview_writer.md` with the `count_mentions.py` output. It walks frontmatter and reads at most 5 full pages. ### 1.6 Rebuild the index ```bash uv run --script .agents/skills/02-llm-wiki-ingest/scripts/build_index_md.py --wiki-dir wiki- ``` Fix anything it reports as an `ERROR:` (a page with no `type`) before moving on. ### 1.7 Append to the log ```markdown ## ingest | - Ingested N notes from ``; skipped M already present () - Source pages: - New pages: (concept, 2 sources), ... - Updated pages: (2 → 3 sources), ... - Waiting at 1 mention: , ``` ### 1.8 Report back New, skipped and refreshed counts by origin, the pages your agents wrote, the wiki path, and one line telling the user to open `wiki-/wiki/index.md` in Obsidian. If any slug is sitting at exactly 1 mention, name a couple — that is the wiki's own to-do list. --- # Query path Read-only. The only file query mode writes in this layer is `log.md`. - **Q.1 — Locate** the wiki (Step 0). - **Q.2 — Load the map.** Read `wiki/index.md`, then the one or two `wiki//index.md` files that look relevant. Each bullet is a `title — description`; that is enough to choose. - **Q.3 — Read 1–3 pages.** Prefer entity/concept pages (already synthesized) over source pages (single-note views). For "how does this code work" questions, start at `wiki/repos//ARCHITECTURE.md` — it exists so nobody has to read the clone. - **Q.4 — Escalate only if you must.** If the wiki pages genuinely do not answer the question, read the specific `raw/` file a source page points to. Never bulk read `raw/`. For a repo page that means specific files under its `clone_path`; clones are never committed, so if it is missing, clone it yourself with `clone_repo.py` (same call as in 1.2) — never ask the user to. If the wiki simply does not know, **say so** — a wiki that hallucinates coverage is worse than an empty one. - **Q.5 — Answer** with `[[wikilinks]]` to every page you used, and end with a `Pages used:` list. Distinguish what the notes claim from what you inferred. - **Q.6 — Log it**: ```markdown ## query | - question: "" - pages used: () - gap noticed: ``` --- ## Harness notes This skill is written for any agent harness. It needs five capabilities: | Capability | Used for | |---|---| | read a file | raw notes, wiki pages | | write a file | every page you create | | run a shell command | `cp`, `ls`, `mkdir`, the scripts | | ask the user | the wiki slug, which wiki to use | | spawn a subagent | every page written in this layer | In Claude Code those are `Read`, `Write`, `Bash`, `AskUserQuestion` and the Task/Agent tool. Nothing in this skill pins a model or a tool name — the agent files state a model *preference* in prose, and any harness that can run a sub-conversation with a prompt can run them. ## Agent reference Read the file, pass its body as the prompt, collect one JSON receipt. | Agent | Reads | Writes | |---|---|---| | `agents/source_writer.md` | one `raw/` file — the only agent that may | one `wiki/sources/.md` | | `agents/repo_writer.md` | one clone under `raw/repos/` | one page under `wiki/repos//` | | `agents/page_writer.md` | the source-like pages for one slug | one `wiki/entities/` or `wiki/concepts/` page | | `agents/overview_writer.md` | frontmatter, plus ≤5 full pages | `wiki/overview.md` | ## Script reference All five scripts are PEP 723 single files — `uv` installs their dependencies on the first run. You run them; the user never does. Paths below are relative to the layer directory. | Script | Call | Does | |---|---|---| | `scripts/count_mentions.py` | `uv run --script .agents/skills/02-llm-wiki-ingest/scripts/count_mentions.py --wiki-dir wiki-` | Frontmatter walk over source-like pages → `{slug: [pages]}` + the `qualifying` list at ≥2. JSON on stdout, table on stderr. | | `scripts/build_index_md.py` | `uv run --script .agents/skills/02-llm-wiki-ingest/scripts/build_index_md.py --wiki-dir wiki-` | Regenerates `wiki/index.md` and each `wiki//index.md` from frontmatter, then checks OKF conformance. Deterministic: same input, same bytes. | | `scripts/clone_repo.py` | `--repo --wiki-dir wiki-` | Shallow-clones into `raw/repos/.github--/`, or refreshes it. JSON receipt with `commit_sha` and `action`. | | `scripts/fetch_article.py` | `--url --wiki-dir wiki-` | `curl` → body isolation → markdown at `raw/article-.md`. Warns when the body looks like a paywall. | | `scripts/fetch_youtube.py` | `--url --wiki-dir wiki-` | Skeleton. Raises `NotImplementedError` — the workshop exercise in `SOURCES.md`. | No script writes wiki content. They fetch, they count, they render — every word in `wiki/` is written by an agent.