# Wiki Ingest > Ingest documents into the Obsidian wiki by distilling their knowledge into interconnected wiki pages. Use this skill whenever the user wants to add new sources to their wiki, process a document or directory, import articles, papers, or notes into their knowledge base, or says things like "add this to the wiki", "process these docs", "ingest this folder". Also triggers when the user drops a file and wants it incorporated into their existing knowledge base. Also handles raw mode: "process my drafts", "promote my raw pages", or any reference to the _raw/ staging directory. - Skill: `mangowhoiscloud/wiki-ingest` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add mangowhoiscloud/wiki-ingest` - Raw SKILL.md: https://api.skillmd.com/api/skills/mangowhoiscloud/wiki-ingest/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Research & Search - Author: mangowhoiscloud (https://skillmd.com/u/mangowhoiscloud) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/mangowhoiscloud/wiki-ingest --- # Obsidian Ingest — Document Distillation You are ingesting source documents into an Obsidian wiki. Your job is not to summarize — it is to **distill and integrate** knowledge across the entire wiki. ## Before You Start 1. Read `.env` to get `OBSIDIAN_VAULT_PATH` and `OBSIDIAN_SOURCES_DIR` 2. Read `.manifest.json` at the vault root to check what's already been ingested 3. Read `index.md` to understand current wiki content 4. Read `log.md` to understand recent activity ## Language policy — wiki content is English-only Distilled wiki pages are written in **English**, regardless of the source document's language. Translate Korean / Japanese / Chinese sources into English when distilling; keep at most one short verbatim quote per page (in source language with an English inline gloss when non-English) so attribution stays honest. Reasons: - The Bagelcode hiring panel + future open-source contributors are the wiki's audience; English is the lowest-friction reading surface for both. - A wiki rebuild / lint / cross-link pass operates on English token boundaries; mixed-language pages create false positives in `wiki-lint` and miss hits in `wiki-query`. - Crumb's root `CLAUDE.md` mandates English for the codebase + commits + PRs (the wiki is a codebase artifact, not chat). ## Ingest Modes This skill supports three modes. Ask the user or infer from context: ### Append Mode (default) Only ingest sources that are **new or modified** since last ingest. Check the manifest using both timestamp **and content hash**: - If a source path is not in `.manifest.json` → it's new, ingest it - If a source path is in `.manifest.json`: - Compute the file's SHA-256 hash: `sha256sum ` (or `shasum -a 256 ` on macOS) - If the hash matches `content_hash` in the manifest → **skip it**, even if the modification time differs (file was touched but content is identical — git checkout, copy, NFS timestamp drift) - If the hash differs → it's genuinely modified, re-ingest it - If a source path is in `.manifest.json` and has no `content_hash` (older entry) → fall back to mtime comparison as before This is the right choice most of the time. It's fast and avoids redundant work even when timestamps are unreliable. ### Full Mode Ingest everything regardless of manifest state. Use when: - The user explicitly asks for a full ingest - The manifest is missing or corrupted - After a `wiki-rebuild` has cleared the vault ### Raw Mode Process draft pages from the `_raw/` staging directory inside the vault. Use when: - The user says "process my drafts", "promote my raw pages", or drops files into `_raw/` - After a paste-heavy session where notes were captured quickly without structure In raw mode, each file in `OBSIDIAN_VAULT_PATH/_raw/` (or `OBSIDIAN_RAW_DIR`) is treated as a source. After promoting a file to a proper wiki page, **delete the original from `_raw/`**. Never leave promoted files in `_raw/` — they'll be double-processed on the next run. ## The Ingest Process ### Step 1: Read the Source Read the document(s) the user wants to ingest. In append mode, skip files the manifest says are already ingested and unchanged. Supported formats: - Markdown (`.md`) — read directly - Text (`.txt`) — read directly - PDF (`.pdf`) — use the Read tool with page ranges - Web clippings — markdown files from Obsidian Web Clipper - **Images** (`.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`) — *requires a vision-capable model*. Use the Read tool, which renders the image into your context. Treat screenshots, whiteboard photos, diagrams, and slide captures as first-class sources. If your model doesn't support vision, skip image sources and tell the user which files were skipped so they can re-run with a vision-capable model. Note the source path — you'll need it for provenance tracking. ### Multimodal branch (images) When the source is an image, your extraction job is interpretive — you're reading visual content, not text. Walk the image methodically: 1. **Transcribe** any visible text verbatim (UI labels, slide bullets, whiteboard handwriting, code snippets in screenshots). This is the only *extracted* content from an image. 2. **Describe structure** — for diagrams, list the boxes/nodes and the arrows/edges. For screenshots, name the app or context if recognizable. 3. **Extract concepts** — what is the image *about*? What ideas, entities, or relationships does it convey? Most of this is `^[inferred]`. 4. **Note ambiguity** — handwriting you can't read, arrows whose direction is unclear, cropped content. Use `^[ambiguous]` and call it out. Vision is interpretive by nature, so image-derived pages will skew heavily toward `^[inferred]`. That's expected — the provenance markers exist precisely to surface this. Don't pretend an image's "meaning" was extracted when you really inferred it. For PDFs that are mostly images (scanned docs, slide decks exported to PDF), use `Read pages: "N"` to pull specific pages and treat each page as an image source. ### Step 1b: QMD Source Discovery (optional — requires `QMD_PAPERS_COLLECTION` in `.env`) **GUARD: If `$QMD_PAPERS_COLLECTION` is empty or unset, skip this entire step and proceed to Step 2.** > **No QMD?** Skip this step entirely. Use `Grep` in Step 4 to check for existing pages on the same topic before creating new ones. See `.env.example` for QMD setup instructions. When `QMD_PAPERS_COLLECTION` is set: Before extracting knowledge from a document, check whether related papers are already indexed that could enrich the page you're about to write: ``` mcp__qmd__query: collection: # e.g. "papers" intent: searches: - type: vec # semantic — finds papers on the same topic even with different vocabulary query: - type: lex # keyword — finds papers citing the same methods, tools, or authors query: ``` Use the returned snippets to: 1. **Surface related papers** you may not have thought to link — add them as cross-references in the wiki page 2. **Identify recurring themes** across the corpus — these deserve their own concept pages 3. **Find contradictions** between this source and indexed papers — flag with `^[ambiguous]` 4. **Avoid duplicate pages** — if the corpus already covers this concept heavily, merge rather than create If the QMD results show that 3+ papers touch the same concept, that concept almost certainly warrants a global `concepts/` page. **Skip this step** if `QMD_PAPERS_COLLECTION` is not set. ### Step 2: Extract Knowledge From the source, identify: - **Key concepts** that deserve their own page or belong on an existing one - **Entities** (people, tools, projects, organizations) mentioned - **Claims** that can be attributed to the source - **Relationships** between concepts (what connects to what) - **Open questions** the source raises but doesn't answer **Track provenance per claim as you go.** For each claim you extract, mentally tag it as: - *Extracted* — the source explicitly states this - *Inferred* — you're generalizing across sources, drawing an implication, or filling a gap - *Ambiguous* — sources disagree, or the source is vague You'll apply markers in Step 5. Don't conflate these — the wiki's value depends on the user being able to tell signal from synthesis. ### Step 3: Determine Project Scope If the source belongs to a specific project: - Place project-specific knowledge under `projects///` - Place general knowledge in global category directories - Create or update the project overview at `projects//.md` (named after the project — never `_project.md`, as Obsidian uses filenames as graph node labels) If the source is not project-specific, put everything in global categories. ### Step 4: Plan Updates Before writing anything, plan which pages to update or create. Aim for 10-15 pages per ingest. For each: - Does this page already exist? (Check `index.md` and use Glob to search `OBSIDIAN_VAULT_PATH`) - If it exists, what new information does this source add? - If it's new, which category does it belong in? - What `[[wikilinks]]` should connect it to existing pages? ### Step 5: Write/Update Pages For each page in your plan: **If creating a new page:** - Use the page template from the llm-wiki skill (frontmatter + sections) - Place in the correct category directory - Add `[[wikilinks]]` to at least 2-3 existing pages - Include the source in the `sources` frontmatter field **If updating an existing page:** - Read the current page first - Merge new information — don't just append - Update the `updated` timestamp in frontmatter - Add the new source to the `sources` list - Resolve any contradictions between old and new information (note them if unresolvable) **Write a `summary:` frontmatter field** on every new page (1–2 sentences, ≤200 characters) answering "what is this page about?" for a reader who hasn't opened it. When updating an existing page whose meaning has shifted, rewrite the summary to match the new content. This field is what `wiki-query`'s cheap retrieval path reads — a missing or stale summary forces expensive full-page reads. **Apply provenance markers** per the convention in `llm-wiki` (Provenance Markers section): - Inferred claims get a trailing `^[inferred]` - Ambiguous/contested claims get a trailing `^[ambiguous]` - Extracted claims need no marker - After writing the page, count rough fractions and write them to a `provenance:` frontmatter block (extracted/inferred/ambiguous summing to ~1.0). When updating an existing page, recompute and update the block. ### Step 6: Update Cross-References After writing pages, check that wikilinks work in both directions. If page A links to page B, consider whether page B should also link back to page A. ### Step 7: Update Manifest and Special Files **`.manifest.json`** — For each source file ingested, add or update its entry: ```json { "ingested_at": "TIMESTAMP", "size_bytes": FILE_SIZE, "modified_at": FILE_MTIME, "content_hash": "sha256:<64-char-hex>", "source_type": "document", // or "image" for png/jpg/webp/gif and image-only PDFs "project": "project-name-or-null", "pages_created": ["list/of/pages.md"], "pages_updated": ["list/of/pages.md"] } ``` `content_hash` is the SHA-256 of the file contents at ingest time. Always write it — it's the primary skip signal on subsequent runs. Also update `stats.total_sources_ingested` and `stats.total_pages`. If the manifest doesn't exist yet, create it with `version: 1`. **`index.md`** — Add entries for any new pages, update summaries for modified pages. **`log.md`** — Append an entry: ``` - [TIMESTAMP] INGEST source="path/to/source" pages_updated=N pages_created=M mode=append|full ``` ## Handling Multiple Sources When ingesting a directory, process sources one at a time but maintain a running awareness of the full batch. Later sources may strengthen or contradict earlier ones — that's fine, just update pages as you go. ## Quality Checklist After ingesting, verify: - [ ] Every new page has frontmatter with title, category, tags, sources - [ ] Every new page has at least 2 wikilinks to existing pages - [ ] No orphaned pages (pages with zero incoming links) - [ ] `index.md` reflects all changes - [ ] `log.md` has the ingest entry - [ ] Source attribution is present for every new claim - [ ] Inferred and ambiguous claims are marked with `^[inferred]` / `^[ambiguous]`; `provenance:` frontmatter block is present on new and updated pages - [ ] Every new/updated page has a `summary:` frontmatter field (1–2 sentences, ≤200 chars) ## Reference Read `references/ingest-prompts.md` for the LLM prompt templates used during extraction.