# LLM Knowledge Base > Use when ingesting any raw file into the wiki. Covers what the AGENTS.md compile checklist steps actually mean, image handling, concept extraction rules, and when to apply which long-doc strategy. - Skill: `tomevault-io/llm-knowledge-base` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/llm-knowledge-base` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/llm-knowledge-base/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/llm-knowledge-base --- # Skill: compile-ingest Follow the compile checklist in AGENTS.md. This skill fills in the *what* behind each step. --- ## fetch-url: command When you receive `fetch-url: [name]`: ```bash ./tools/fetch-url.sh "" [name] ``` The script auto-detects static vs JS page: - Static page (≥200 words) → uses `requests` + `markdownify` — fast, token-efficient - JS-rendered page (<200 words) → falls back to `r.jina.ai` automatically - Force JS: `./tools/fetch-url.sh "" --force-jina` After the script saves `raw/articles/.md`, continue with the normal compile checklist from Step 3. --- ## Step 3 — Image Handling (before reading) Check for external images first: `grep -c 'https://' "raw/articles/file.md"` If count > 0 → run `./tools/fetch-images.sh "raw/articles/file.md"` before reading. Without this, external images show as broken URLs — Claude cannot see them and will write wiki content with missing visual context. **Image triage when writing wiki:** - HIGH VALUE (diagrams, flowcharts, architecture, data charts) → describe in detail + embed `![name](../../raw/images/name.png)` - LOW VALUE (banners, avatars, stock photos) → ignore completely, do not mention --- ## Step 4 — Core Cognitive ### Summary format (`wiki/summaries/.md`) Required sections in this order: 1. **Executive Summary** — 1–2 paragraphs: context, problem, main thesis 2. **Deep Analysis** — logical breakdown with sub-headings 3. **Key Insights** — 2–5 bullets of the most novel/counter-intuitive findings 4. *(optional)* **Limitations & Open Questions** 5. **Concepts** + **Relations** — wikilinks at the end Write to be understood, not just to record. Explain *why* something matters, not just *what* it is. ### Concept extraction (`wiki/concepts/.md`) **Hard limit: 1–3 MACRO concepts per document.** Minor terms and sub-concepts must be grouped under a parent macro — do not give them their own files. Required frontmatter: `domain:` field is non-negotiable. Use the domain MOC slug (`ai`, `product`, `meta`...). If no MOC exists yet: use a short slug, lint will detect it. **Compiled Truth + Timeline format** — every concept file ends with: ```markdown --- - YYYY-MM-DD: Source — one line on how/why compiled or updated ``` - Above `---` = compiled truth (rewrite when evidence changes) - Below `---` = timeline (append-only, never edit existing entries, only add new) ### Long document strategy Run `./tools/scan.sh --info ` → check word count + section map → pick strategy: | Words | Strategy | Notes | |-------|----------|-------| | < 4K | **STUFFING** | Read entire file at once | | 4K–10K | **REFINE** | Sequential reads by section, running summary | | 10K–25K | **MAP-REDUCE** | Parallel agents per section (see below) | | > 25K | **HIERARCHICAL** | Per-section summaries → synthesize | **MAP-REDUCE — correct flow** (do NOT chunk by line count): 1. `./tools/scan.sh --info ` — output now includes section map with exact line boundaries 2. Chunk **by section boundaries** from the map, skip References/Appendix/Proofs 3. Spawn agents in parallel, one per logical section group 4. Each agent: read its lines → produce structured notes (NOT a full summary) 5. REDUCE: combine all notes → write final summary + concepts **Cost trade-off**: MAP-REDUCE costs ~3–4x more tokens than sequential. Use only when context overflow is a real risk (> ~15K words in dense technical text). For most papers and articles, sequential read is faster and cheaper. --- ## Step 4.5 — Verify before finalizing - [ ] Summary has all required sections? - [ ] Concepts ≤ 3 macro, each with `domain:`? - [ ] All `[[wikilinks]]` point to files that exist (or will exist after this compile)? - [ ] No unexplained jargon stacking? If any check fails → fix first, then proceed to Step 5. --- ## Step 5 — finalize-compile.sh ```bash ./tools/finalize-compile.sh "raw/path/to/file" "One-line key insight" --model claude-sonnet-4-6 ``` `--model` is required — pass your actual model ID. Supported IDs: see `tools/metrics.py`. If no domain MOC exists yet and ≥10 concepts exist → manually create MOC following `wiki/domains/_about-domains.md`. --- ## Gotchas **Concept explosion** — The most common mistake. After reading an interesting document, the instinct is to create a concept for every interesting term. Resist. Ask: "Is this a MACRO idea that stands on its own, or a sub-concept that belongs under an existing concept?" When in doubt, group it. **Missing `--model` flag** — `finalize-compile.sh` runs silently without it but cost tracking is broken. The flag is not optional; it's required for metrics. **Editing TIMELINE entries** — The `` section is append-only. Never edit existing dated entries, even to fix typos. Only add new dated lines at the bottom. **Skipping fetch-images.sh** — If you read a raw `.md` file before downloading its images, external URLs appear as broken links. You'll write wiki content without seeing the actual diagrams. Always grep for `https://` first. **Choosing strategy without `scan.sh --info`** — Don't guess the word count. A "short article" might be 12,000 words. Running `scan.sh --info` takes 2 seconds and determines the right strategy. **Concept file > 150 lines** — If you're going over, the concept is doing too much. Split into parent + sub-concept, or move examples to a summary instead. **Not verifying `[[wikilinks]]` before finalizing** — Orphaned links will fail lint. Check that every `[[linked-concept]]` either exists already or will be created in this same compile session. --- > Source: [hoadoan1997/llm-knowledge-base](https://github.com/hoadoan1997/llm-knowledge-base) — distributed by [TomeVault](https://tomevault.io).