How do I install the Ingest skill?

Run npx skillmds@latest add zedarvates/ingest in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Ingest safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Ingest?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Ingest free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Ingest?

zedarvates (@zedarvates) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Ingest by zedarvates — Agent Skill

Name: Ingest
Author: zedarvates

Ingest

Local-first web scraping and source ingestion — fetch a URL, extract clean text locally (0 cloud tokens), optionally structure it with a local model, and store it in a Qdrant collection (the second-brain "foundation") for later recall. Use for /web-scraping, /ingest-source, building a knowledge foundation, or recalling past ingested sources. Web fetch + Qdrant over stdlib HTTP.

zedarvates 8ecbb42 5 files · 19.8 KB Updated Sep 17, 2026

File contents

ingest — scrape the web & build a knowledge foundation, locally

Scraping and ingesting are extraction/transformation — keep them off the cloud.

python -m skills.ingest.cli scrape  https://example.com --structure   # local summary+entities
python -m skills.ingest.cli ingest  https://example.com --collection foundation
python -m skills.ingest.cli ingest  ./notes.md --file
python -m skills.ingest.cli search  "topic"  --collection foundation

scrape — fetch (browser UA) + stdlib HTML→text (drops script/style, keeps title). --structure asks a LOCAL model for a 3-bullet summary + key entities. 0 cloud tokens.
ingest — scrape (or a file/raw text) → reflect locally → upsert into Qdrant (192.168.1.47:6333 by default). Builds the "foundation"/historical store.
search — recall from a collection.

Embeddings: auto-resolve a local /v1/embeddings endpoint from the backend registry (any reachable backend exposing an embedding model → real semantic vectors), falling back to a deterministic hash n-gram (256-dim) when none is available — so it always works, 0 cloud tokens either way. Override with --embed-url/--embed-model. The result reports embed: endpoint|hash. Degrades gracefully when Qdrant is down (scrape still works).

python -m skills.ingest.cli ingest https://example.com --embed-url http://127.0.0.1:1234/v1/embeddings

Exposed via [[llm_mcp]] as scrape and ingest_source. Related: [[hermes-second-brain]] (the foundation concept), media_loader, [[auto_router]].

---name: ingestdescription: Local-first web scraping and source ingestion — fetch a URL, extract clean text locally (0 cloud tokens), optionally structure it with a local model, and store it in a Qdrant collection (the second-brain "foundation") for later recall. Use for /web-scraping, /ingest-source, building a knowledge foundation, or recalling past ingested sources. Web fetch + Qdrant over stdlib HTTP.---# ingest — scrape the web & build a knowledge foundation, locallyScraping and ingesting are extraction/transformation — keep them off the cloud.```bashpython -m skills.ingest.cli scrape https://example.com --structure # local summary+entitiespython -m skills.ingest.cli ingest https://example.com --collection foundationpython -m skills.ingest.cli ingest ./notes.md --filepython -m skills.ingest.cli search "topic" --collection foundation```- **scrape** — fetch (browser UA) + stdlib HTML→text (drops script/style, keeps title). `--structure` asks a LOCAL model for a 3-bullet summary + key entities. **0 cloud tokens.**- **ingest** — scrape (or a file/raw text) → reflect locally → upsert into Qdrant (`192.168.1.47:6333` by default). Builds the "foundation"/historical store.- **search** — recall from a collection.Embeddings: **auto-resolve a local `/v1/embeddings` endpoint** from the backendregistry (any reachable backend exposing an embedding model → real semanticvectors), falling back to a deterministic hash n-gram (256-dim) when none isavailable — so it always works, **0 cloud tokens** either way. Override with`--embed-url`/`--embed-model`. The result reports `embed: endpoint|hash`. Degradesgracefully when Qdrant is down (scrape still works).```bashpython -m skills.ingest.cli ingest https://example.com --embed-url http://127.0.0.1:1234/v1/embeddings```Exposed via [[llm_mcp]] as `scrape` and `ingest_source`. Related:[[hermes-second-brain]] (the foundation concept), `media_loader`, [[auto_router]].

Frequently asked questions