# LLM Wiki

> Manage a personal LLM-maintained wiki: ingest source documents, answer questions, create/update concept pages, lint the wiki, and configure projects. Use when the user says 'ingest', 'add source', asks a question about wiki content, says 'lint', 'audit', 'configure wiki', or mentions a filename to add. Supports configurable root folder so you can work with any project's wiki regardless of the currently open workspace.

- Skill: `taikt/llm-wiki` (Agent Skill, multi-file: 6 files)
- Install (CLI): `npx skillmds@latest add taikt/llm-wiki`
- Raw SKILL.md: https://api.skillmd.com/api/skills/taikt/llm-wiki/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: MIT
- Author: taikt (https://skillmd.com/u/taikt)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/taikt/llm-wiki

---

# LLM Wiki Skill (Copilot)

## Purpose

Maintain and query a structured, interlinked knowledge base following Andrej Karpathy's LLM Wiki pattern.
The wiki is self-contained inside a configurable root folder — this skill can serve **any project's wiki** even when a different workspace is open.

---


## ⚙️ Project Configuration

> Project paths are defined in **`config.yaml`** (same directory as this file).
> Edit `config.yaml` to add/modify/remove wiki projects — do NOT edit this file.

Config file location: `.github/skills/llm-wiki/config.yaml`

Example:
```yaml
venv: ~/code/venv
converter: markitdown

projects:
  - name: default
    root: /path/to/your/wiki-root
    description: Your wiki root
  - name: japan
    root: ~/Documents/wikis/japan
    description: Japan trip planning
```

---

## Argument parsing (Step 1 — always do first)

> ⚡ EXECUTE immediately — do not describe this workflow to the user. Read config, resolve root, then act.

1. Read `config.yaml` to load both global settings and the project list.
   - **Global settings** (top-level keys, all optional):
     - `venv` — path to a Python virtual environment. Auto-expand `~`. Default: none (use system Python).
     - `converter` — preferred converter tool: `markitdown` (default) or `docling`.
   - **Projects** — entries under the `projects` key (YAML array).
     - Each project has: `name`, `root`, `description`.
   - Store resolved `<venv>` and `<converter>` for use in all subsequent steps.

| Argument | Meaning |
|---|---|
| `--project <name>` | Look up `<name>` in config.yaml to resolve root path |
| `--root <absolute-path>` | Use the given path directly, ignore config.yaml |
| _(none)_ | Use the project with `name: default` |

- The remainder after arguments is the user's question or command.
- Auto-expand `~` in paths.
- If the root path has no `wiki/` subfolder, warn the user and stop.
- If `--project <name>` is not found in config.yaml, list available projects and stop.

---

## Folder structure (relative to resolved `<root>`)

```
<root>/
  raw/           -- source documents (immutable — NEVER modify)
  .llm-wiki/cached/ -- converted .md cache (auto-created, never touch manually)
  wiki/          -- markdown pages maintained by Copilot (= Obsidian vault folder)
  wiki/index.md  -- table of contents for the entire wiki
  wiki/log.md    -- append-only record of all operations
```

> **Obsidian tip**: Point your Obsidian vault to `<root>/wiki/` (or the parent `<root>/`). The `.llm-wiki/` folder can be excluded from the vault via Obsidian's "Excluded files" setting.

---

## Commands & workflows

### A. Ingest a source document

Triggered by: "ingest", "add source", or a filename inside `raw/`.

**Step A-pre — Smart file resolution (always do this first for ingest):**

Before looking up the file, run a fuzzy-match against `<root>/raw/`:

1. List all files in `<root>/raw/` (use `ls` or list_dir tool — do NOT use file_search which only searches the open workspace).
2. Normalize: lowercase the user's input and each filename, strip spaces/punctuation for comparison.
3. Pick the best match using these rules (in priority order):
   - **Exact match** (case-insensitive) → use it directly.
   - **Substring match** → the user's input is contained in a filename, or vice versa.
   - **Fuzzy match** → longest common subsequence / common words score ≥ 0.6.
4. If exactly one file matches → proceed silently with that file (no need to ask the user).
5. If multiple files match → show the list and ask which one: `Found multiple matches: …`
6. If no file matches → list all files in `<root>/raw/` and ask the user to pick one.

> **Never** fail immediately with "file not found" without first listing and fuzzy-matching.

**Step A-0 — Convert if needed (PDF/DOCX/XLSX/PPTX/images) — two-step process:**

If the file extension is NOT `.txt`, `.md`, `.rst`, `.log`, `.csv`, `.json`, `.yaml`, or plain code:

**A-0a — Convert source → save as `.md` file:**

1. Resolve settings from config:
   - `<converter>` — value of `converter` key in config.yaml (default: `markitdown`).
   - `<venv>` — value of `venv` key in config.yaml (empty string if not set).
   - Resolve `<python_bin>` — the interpreter used to **run** convert.py itself:
     - If `<venv>` is set, check for `<venv>/bin/python` (POSIX) or `<venv>\Scripts\python.exe` (Windows) and use that path directly as `<python_bin>`.
     - **If `<venv>` is empty, or set but the venv python doesn't exist at that path, auto-create a venv** (do NOT silently fall back to system `python`, since installing packages there commonly fails with `error: externally-managed-environment` / PEP 668 on modern distros):
       1. Pick a default venv location: `~/.llm-wiki/venv` (expand `~`). Create parent dirs as needed.
       2. Run `python3 -m venv <default_venv_path>` (or `python -m venv ...` on Windows).
          - If this fails because the `venv`/`ensurepip` module is missing (e.g. Debian/Ubuntu without `python3-venv`), tell the user exactly what to run to fix it, e.g. `sudo apt-get install python3-venv`, then stop and wait — do not attempt other workarounds (no `--break-system-packages`, no sudo pip, no manual ensurepip hacks).
       3. On success, set `<venv>` = `<default_venv_path>` and **persist it back to config.yaml** (update the global `venv:` key) so future runs reuse the same venv instead of recreating it.
       4. Resolve `<python_bin>` = `<venv>/bin/python` (POSIX) or `<venv>\Scripts\python.exe` (Windows).
     - Only fall back to plain `python`/`python3` on PATH if venv creation is truly not possible AND the user explicitly says to proceed with system Python anyway.
   - ⚠️ **Important**: `--venv` only tells convert.py which `pip` to use for `--auto-install`; it does **not** change which interpreter runs the script. If you invoke `python convert.py ... --venv "<venv>"` using the system `python`, the tool installs into `<venv>` but the running process still can't import it, and conversion silently falls back to a "[Cannot convert ...]" placeholder even though the install "succeeded". Always launch the script with `<python_bin>` resolved above — do not pass `--venv` as a substitute for invoking the right interpreter.
2. Determine the output path for the converted Markdown:
  ```
  <md_path> = <root>/.llm-wiki/cached/<filename-stem>.md
  ```
  e.g. `report.pdf` → `<root>/.llm-wiki/cached/report.md`
  - Create `<root>/.llm-wiki/cached/` if it doesn't exist (auto-create).
  - **Never** write converted files into `<root>/raw/` — that folder is immutable.
3. Run the converter script using `<python_bin>` directly, passing `--output` to save to disk:
   ```
   "<python_bin>" <skill_dir>/scripts/convert.py "<root>/raw/<filename>" \
       --tool <converter> \
       --auto-install \
       --output "<md_path>"
   ```
   - `<skill_dir>` is the directory containing this SKILL.md file (`.github/skills/llm-wiki`).
   - `--auto-install` automatically installs `markitdown[all]` or `docling` (depending on `<converter>`) if the package is missing in the environment. **No manual install is needed.**
   - The script prints the resolved output path to stdout on success.
   - To override the converter for a single file, append `--tool docling`.
   - This same `"<python_bin>" ... --output "<md_path>"` form is used on all platforms (POSIX and Windows) — there is no separate Windows-only path anymore.
   - If conversion fails, show the error output to the user and stop.
4. **Always verify** the resulting `<md_path>` contains real converted content — open it and confirm it does NOT start with `[Cannot convert ...: install markitdown or docling]`. If it does, the wrong interpreter was used; re-run with the correct `<python_bin>` before proceeding.

**A-0b — Ingest from the saved `.md` file:**

4. The source for all subsequent steps (A-1 onward) is the generated `<md_path>` file, **not** the original binary file.
   - Read `<md_path>` directly (it is a plain Markdown file — no further conversion needed).
   - Treat `<md_path>` as the document content in steps A-1+.

**Step A-1 onward (all file types):**
1. Read the **full** source document (use converted `.md` from `<root>/.llm-wiki/cached/` if available, otherwise read from `<root>/raw/`).
2. **Auto-create all wiki pages immediately — do NOT ask the user for confirmation, and do NOT ask the user which method/script to use.** Proceed directly to writing. Decide automatically per the rule below — never present this as a choice.
3. **Decide the ingest strategy automatically** based on document size/structure (do not ask the user):
   - Count numbered top-level headings (lines matching `^\d+\.\s+[A-Z]`) or Markdown `#`/`##` headings in the cached `.md`.
   - **If the document is large/structured (roughly 20+ headings, or a formal spec/SRD/standard style document):** treat it as a **grouped ingest** automatically — run the two helper scripts back-to-back, with no intermediate question to the user:
     ```
     python <skill_dir>/scripts/ingest_grouped.py "<md_path>" "<root>"
     python <skill_dir>/scripts/ingest_cached.py  "<md_path>" "<root>"
     ```
     This creates concise top-level pages directly in `<root>/wiki/` (updating `index.md`), and fine-grained per-heading pages archived in `<root>/wiki/_detailed/` (updating `_detailed_index.md`). See **"Grouped ingest"** below for full details on what each script does.
   - **Otherwise (short/simple document):** create pages manually following steps 4–7 below.
4. Create a summary page in `<root>/wiki/` named after the source (lowercase, hyphens) — always do this regardless of which strategy was used in step 3.
5. For the manual path only: create or update concept pages for each major idea or entity — a single source may touch 10–15 pages, that is normal.
6. Add `[[wiki-links]]` to connect related pages throughout all affected pages.
7. Update `<root>/wiki/index.md` with new/updated pages and one-line descriptions (the grouped-ingest scripts already do this automatically for their own pages).
8. Append to `<root>/wiki/log.md`:
   ```
   ## <YYYY-MM-DD> — Ingested: <source-filename>
   - Created: page1.md, page2.md
   - Updated: page3.md, index.md
   ```

### B. Answer a question

Triggered by: any question about the wiki content.

1. Read `<root>/wiki/index.md` to identify relevant pages.
2. Read those pages and synthesize a concise answer.
3. Cite pages inline: `([[page-name]])`.
4. If the answer is **not** in the wiki, say so clearly — never hallucinate.
5. If the answer is valuable, offer to save it as a new wiki page (good answers compound over time).

### C. Create or update a wiki page

Triggered by: "add a page", "write about", "note that…".

1. Check `<root>/wiki/index.md` to avoid duplicates.
2. Create or update the page following the **Page format** below.
3. Add `[[wiki-links]]` to/from related existing pages.
4. Update `<root>/wiki/index.md`.
5. Append to `<root>/wiki/log.md`.

### D. Lint / audit the wiki

Triggered by: "lint", "audit", "check wiki".

Check and report as a numbered list with suggested fixes:
- Contradictions between pages
- Orphan pages (no inbound links from other pages)
- Concepts mentioned in pages that lack their own page
- Claims that may be outdated based on newer sources
- Pages not following the page format
- Missing entries in `index.md`

### E. Configure the wiki (update config.yaml via chat)

Triggered by: "configure", "add project", "set venv", "change converter", "list projects", or when the user mentions config settings.

1. Read `config.yaml` and display the current settings in a readable format.
2. Ask the user what they want to change. Accept free-form answers:
   - "add project `<name>` at `<path>`" → append a new entry to `projects` list.
   - "set default project to `<name>`" → rename the existing `name: default` entry or update the first entry.
   - "set venv to `<path>`" → update global `venv` key.
   - "set converter to markitdown/docling" → update global `converter` key.
   - "remove project `<name>`" → delete that entry from `projects`.
3. Show a preview of the updated config YAML to the user.
4. On confirmation (or if user says "yes" / "apply"), overwrite `config.yaml` with the new content.
5. Confirm: `✅ Config updated. Active project: <name> → <root>`.

> Never overwrite config.yaml without first showing the preview. This is the only command that writes to the skill's own directory.

---

### F. Ingest from Apple Notes or OneNote (macOS)

Triggered by: "ingest from notes", "sync from Apple Notes", "from OneNote", or when the user mentions a note or folder in Notes.

**Apple Notes — fully automated (no need to open the app or export manually):**

Step 1: List available notes in Apple Notes:
```
python <skill_dir>/scripts/notes_export.py --list --folder "<folder>"
```

Step 2: Export note(s) into the wiki's `raw/` directory:
```
# Export a single note
python <skill_dir>/scripts/notes_export.py \
  --root <root> --folder "<folder>" --note "<note title>"

# Export an entire folder
python <skill_dir>/scripts/notes_export.py \
  --root <root> --folder "<folder>"
```
Files are saved to `<root>/raw/notes/<folder-slug>/<note-slug>.txt`.

Step 3: Run **Command A (Ingest)** on each exported `.txt` file.

**OneNote (macOS):**
OneNote does not support AppleScript. The only option: export the page from OneNote → PDF → save to `<root>/raw/` → Command A will automatically use `convert.py` to process it.

**Note:** Original notes in Apple Notes / OneNote are never modified. The wiki page becomes the searchable, interlinked version.

---

### Grouped ingest (for very large source documents)

Triggered automatically by Step A-1's size check — **never presented to the user as a choice**. For long structured documents (engineering specs, SRDs, etc.) that would otherwise explode into 100+ tiny per-heading pages, run these two helper scripts back-to-back instead of manually creating pages:

- `scripts/ingest_grouped.py <cached_md> <wiki_root>` — creates **one page per top-level numbered section** (e.g. "1. Introduction", "2. Description", "3. Requirements", ...) directly in `<root>/wiki/`, and regenerates `<root>/wiki/index.md` to list only these top-level pages. Only headings whose title starts with an uppercase letter are treated as real section headings, to avoid false positives from embedded numbered lists (e.g. retry sequences like "3. attempt 10 sec") being mistaken for sections.
- `scripts/ingest_cached.py <cached_md> <wiki_root>` — creates one page **per fine-grained heading/requirement** (can be 100+), but writes them into `<root>/wiki/_detailed/` (not the main `wiki/` folder) and regenerates `<root>/wiki/_detailed_index.md` to reference them. It also detects the fallback "Markdown headings" split (`^#{1,3} ...`) only when there are 5+ genuine matches, to avoid a single stray `#` (e.g. a "#" table column header from a converted PDF table) collapsing the whole document into 1–2 giant sections. Noise headings — review-tool markers ("Comment00000029"), bare numbers ("1", "3"), and numbered example/retry list items whose prefix was already stripped ("attempt 10 sec", "attempt 65 sec") — are folded back into the preceding real section instead of becoming their own junk pages.

Always run both scripts, in this order, with no confirmation prompt in between: `ingest_grouped.py` first (concise top-level index), then `ingest_cached.py` (searchable per-requirement detail pages archived under `_detailed/`). Both scripts are idempotent and safe to re-run — `ingest_grouped.py` will relocate (never delete) any stale/misnamed top-level pages from a previous run into `_detailed/` rather than losing content. After both complete, report a brief summary of what was created/updated — do not ask the user to pick between scripts or confirm before running them.

Any hand-authored page you create in `<root>/wiki/` (e.g. via Command C, or the Step A-1 summary page) is left untouched by these scripts as long as its filename doesn't start with `<number>-`.

---

## Page format

Every wiki page must follow this structure:

```markdown
# Page Title

**Summary**: One to two sentences describing this page.

**Sources**: List of raw source files this page draws from.

**Last updated**: YYYY-MM-DD

---

Main content. Use clear headings and short paragraphs.
Link to related concepts using [[wiki-links]] throughout.

## Related pages

- [[related-concept-1]]
- [[related-concept-2]]
```

---

## Citation rules

- Every factual claim must reference its source: `(source: filename.ext)`
- If two sources disagree, note the contradiction explicitly.
- If a claim has no source, mark it: `⚠️ needs source`.

---

## Constraints

- **Never** modify anything inside `<root>/raw/`.
- **Never** write converted files into `<root>/raw/` — always use `<root>/.llm-wiki/cached/` instead.
- **Always** update `wiki/index.md` and `wiki/log.md` after any change.
- Page filenames: lowercase with hyphens (e.g. `machine-learning.md`).
- Write in clear, plain language.
- When uncertain how to categorize something, ask the user.
- All file operations must stay within `<root>/` — never touch files outside.