# Wiki Ingestion

> Ingest sources into a Hermes LLM Wiki: classify articles/papers/transcripts, manage the raw inbox, snapshot sources append-only, and handle oversized or unknown files.

- Skill: `tjboudreaux/wiki-ingestion` (Agent Skill)
- Install (CLI): `npx skillmds@latest add tjboudreaux/wiki-ingestion`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tjboudreaux/wiki-ingestion/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: MIT
- Author: tjboudreaux (https://skillmd.com/u/tjboudreaux)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tjboudreaux/wiki-ingestion

---


# Hermes Wiki Ingestion

This is the default ingestion skill for Hermes LLM Wikis. It describes how raw
material becomes attributable Wiki Pages. A wiki can override this default with
`hermes wiki skills set ingestion <skill-name>`.

**Media sources** (PDFs, images, audio, video, YouTube, social links) have
their own protocols — load `wiki:wiki-media-ingestion` before ingesting them.

## The pipeline

Every ingest follows the same deterministic flow:

1. **Snapshot** — the source bytes are stored append-only under `raw/` (never
   overwritten; re-ingesting changed content creates a new versioned snapshot
   and flags `drift_detected`).
2. **Classify** — built-in classifiers detect `article`, `paper`, or
   `transcript`. If none match, trusted custom classifiers (see
   `wiki plugins`) get a chance. Anything else lands as `unknown` and is
   retained for review, never silently dropped.
3. **Process** — the processor for the classified label generates or updates
   Wiki Pages (a `source` page plus extracted `concept`/`entity` pages),
   cross-linking existing pages.
4. **Propagate** — index.md sections, log.md attribution rows, the SQLite
   projection, and a git commit are all updated atomically; failures roll back.

## Ingesting one source

```bash
hermes wiki ingest <local-path-or-https-url> [--wiki <slug>] [--author <name>]
```

- Requires a write grant (`HERMES_WIKI=<slug>` or config write grant).
- URLs are fetched and deduplicated by sha256 — unchanged content is a no-op
  (`no change: <source-id>`).
- The 50MB Phase-1 cap applies; larger sources are refused.

## The inbox flow

Drop files into `<wiki-root>/raw/inbox/` (or let monitors deliver them), then:

```bash
hermes wiki inbox                  # list pending files with status + classifier
hermes wiki ingest --inbox         # batch-process everything processable
```

Per-file outcomes:

- `Ingested <name> class=<label>` — pages were generated.
- `Retained <name> class=unknown` — kept in the inbox for a human/agent to
  classify. Override via the dashboard inbox view or the classify API, then
  re-process.
- `Skipped <name> status=oversized` — exceeds the 50MB cap; not processable
  in this phase. Delete it or handle it out-of-band.

Classifier overrides persist in `raw/inbox_status.json` and survive failed
attempts; an override pins the label for the next `ingest --inbox` run.

## Recurring ingestion (monitors)

For sources that update on a schedule, define a portable monitor instead of
ingesting manually:

```bash
hermes wiki monitor --source rss --schedule "0 8 * * *" --prompt "Sweep feeds…"
hermes wiki monitor --setup --yes
```

Monitor sweeps run with `author_kind=cron` and full attribution.

## Custom classification

When the built-ins misclassify a domain-specific format, write a classifier at
`plugins/classifiers/<name>.py` exposing
`classify(source_path) -> ClassLabel | str | None`, then trust it explicitly:

```bash
hermes wiki plugins trust classifier <name>
```

Trust is recorded canonically in `SCHEMA.md` (path + sha256) — editing the file
after trusting silently disables it until re-trusted.

## Synthesis guidance

When expanding ingestion-derived pages (or writing new ones from ingested
sources), apply the synthesis protocol from the `wiki-writing` skill:

- **Page-creation threshold**: derive pages for entities/concepts that appear
  in 2+ sources or are central to this one — never for passing mentions.
- **Drift means review**: when re-ingest reports `drift_detected`, review
  pages whose `sources:` cite the superseded snapshot. Newer content
  generally supersedes; genuine conflicts follow the contradiction protocol
  (`contested: true` + `contradictions:` frontmatter). Raw snapshots stay
  append-only — only wiki pages are updated.

## Rules

- Never write into `raw/` by hand during ingestion — let the pipeline snapshot.
- Never re-classify by renaming files; use the override surfaces.
- Unknown is a valid outcome: retain, review, re-classify. Do not force-fit.
- Check `hermes wiki lint` after large batches; it repairs the projection.