# Wiki Ingest

> Convert URLs, files, or pasted text into typed-knowledge pages with raw provenance. Supports single and batch mode.

- Skill: `karlorz/wiki-ingest` (Agent Skill)
- Install (CLI): `npx skillmds@latest add karlorz/wiki-ingest`
- Raw SKILL.md: https://api.skillmd.com/api/skills/karlorz/wiki-ingest/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: karlorz (https://skillmd.com/u/karlorz)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/karlorz/wiki-ingest

---

# wiki-ingest
## When This Skill Activates
- User shares a URL, paste, or local file to capture in the vault.
- The output target is `entities/`, `concepts/`, `comparisons/`, or `queries/`.
- A vault is resolvable (see step 0).
## Output language
Run `skillwiki lang` at the start. Generate page-body prose, narrative sections, and `--human` summaries in the resolved language. Frontmatter keys, file names, schema headers, index/log structural lines, citation markers, and wikilink slugs MUST stay English.
## Pre-orientation reads (mandatory before any write)
1. `SCHEMA.md`
2. `index.md`
3. Last 20–30 entries of `log.md`
4. (Project context only) `projects/{slug}/README.md` and last ~5 work-item logs.
## Steps (in order — deterministic raw capture and shared publication)
0. **Resolve vault and language.** Run `skillwiki path` (fail if NO_VAULT_CONFIGURED) and `skillwiki lang`. Use the resolved vault path for all writes; use the canonical language for all generated prose.
1. **Guard.** For each URL: run `skillwiki fetch-guard <url>`. If exit ≠ 0, STOP and surface the error. Do not retry.
2. **Fetch.** Use `web_fetch` (or read local file) under Layer 2 controls (the CLI Layer 2 fetcher applies in tests; in skill runtime use `web_fetch` directly and treat any error as STOP).
   - **Portable local-source rule:** follow `using-skillwiki` → Portable Source References. Do not use `source_url: file:///...` as the canonical durable reference; prefer commit-pinned GitHub `blob/<commit>/<path>` when resolvable, else empty `source_url` plus portable repo-relative prose.
3. **Stage pasted text as a file source.** `skillwiki ingest` accepts a readable local source file or HTTP(S) URL; it does **not** accept literal pasted text. For a paste, stage the exact text in a temporary file outside the vault, then pass its path to the normal command:
   ```bash
   skillwiki ingest <staged-paste-path> \
     --vault <resolved-vault> \
     --type <entity|concept|comparison|query> \
     --title "<title>" \
     --tags "<tag1,tag2>" \
     --provenance <research|project>
   ```
   Record the staged path and exact command inputs before execution. If ingestion or typed-page publication fails, retain the staged source and exact command inputs for retry. Remove the staged file only after `skillwiki ingest` exits 0 after typed-page publication; confirm the non-dry-run result reports its raw path, typed path, and publisher operation. Do not stage pasted text inside the vault.
4. **Identity guard.** Before writing raw files, ensure the target raw filename/title, `source_url`, fetched H1/title, and early body subject agree. If `skillwiki ingest` reports `INGEST_VALIDATION_FAILED` with `source identity conflict`, STOP. Do not fix by renaming after the fact; choose the correct title/source pair or ask the user.
5. **Sensitive content guard.** Before writing or filing any vault page, scan the source and generated body for live credentials, access keys, tokens, passwords, cookies, bearer headers, or private keys. Redact generated prose before writing. If the source itself must remain raw and contains a live secret, STOP instead of preserving it.
6. **Feature-detect publication.** Run `skillwiki page publish --help`. If unavailable, fail closed and leave typed output unpublished; update the active SkillWiki CLI/plugin channel first.
7. **Ingest and publish.** Use `skillwiki ingest` for deterministic source capture and typed-page publication. The command writes an immutable raw source first and delegates the typed page, taxonomy, index, and structural log entry to the shared page publisher. Supply the resolved vault, type, title, tags, and provenance through the command options.
8. **Recovery.** Never create the final typed page or edit index.md/log.md directly. A raw-only result after publication failure is valid recovery state. Keep the exact command inputs and retry; do not delete or overwrite the raw source.
9. **Pending-compile path** (raw already exists; clipper / `sources pending`). Do **not** run `skillwiki ingest` again — that would create a second raw object. Interactive sessions only:
   1. `skillwiki sources pending [vault]`
   2. `skillwiki sources compile claim <raw-path> [vault] --reason "<why>"` then the same command with `--write --approve <token>`
   3. Read the raw; compose unpublished drafts **outside** the vault
   4. `skillwiki page publish` each typed page
   5. `skillwiki sources compile published <raw-path> [vault] --pages "<typed/a.md,typed/b.md>" --reason "<why>"` then `--write --approve <token>`
   6. Leave the review `open` unless the human resolves it with `skillwiki sources review <raw-path> --status accepted|needs-fix|dismissed --reason "<why>"`
   Goal, satellite, and headless sessions may list pending / compile status / reviews but must not claim or publish compile turns.
## Provenance defaults
- Default `provenance: research`.
- If cwd is inside `projects/{slug}/`, set `provenance: project` and add `provenance_projects: ["[[slug]]"]`.
## Raw Data Locality
Raw ephemeral data (market feeds, logs, transient JSON) must be written to the **project local** `raw/` directory, NOT the cloud-mounted wiki path. See `references/raw-data-locality.md` for the full pattern.
**Quick rule:**
- Transient data → `~/projects/{slug}/raw/` (local, git-tracked)
- Compound pages → `~/wiki/projects/{slug}/compound/` (cloud, durable)
## Stop conditions
- `fetch-guard` non-zero.
- Fetch timeout / size limit exceeded.
- `INGEST_VALIDATION_FAILED` with `source identity conflict`.
- Source or generated content contains unredacted live credentials or other authenticating secrets.
- `skillwiki page publish --help` is unavailable.
- `skillwiki ingest` returns nonzero; retain any raw-only result for retry.
- sha256 already exists in vault for the same source.
## Forbidden
- Skipping `fetch-guard`.
- Creating a final typed page or editing `index.md`/`log.md` directly.
- Modifying any existing file in `raw/`.
- Writing raw ephemeral data directly to cloud-mounted wiki paths (`~/wiki/`).
- Writing host-local absolute paths as canonical durable source references (see `using-skillwiki` → Portable Source References).
- Writing `[[wikilinks]]` to pages that don't exist in the vault. Before linking, verify the target exists: check `index.md` or `ls` the target directory. If the target doesn't exist yet, use plain text instead of a wikilink.
- Indexing `raw/assets/` binaries with only a `.txt` sidecar. Write a sibling Markdown note that embeds each file with `![[ ]]`.
## Batch Mode
When the user provides multiple sources (a directory of files, a list of URLs, or a multi-document input):
1. **Loop per source.** Execute steps 1–8 for each source individually, using one `skillwiki ingest` command per source.
2. **Fail fast.** If an ingest command returns nonzero, STOP and report the retained raw-only state, if any, with its exact retry inputs.
3. **Deduplication.** Let `skillwiki ingest` preserve immutable raw capture and skip sources whose content is already present.
4. **Progress.** After each source completes, report the raw path, typed path or recovery state, and publisher operation ID.