How do I install the Knowledge Pipeline skill?

Run npx skillmds@latest add s3yed/knowledge-pipeline in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

What does the Knowledge Pipeline skill do?

knowledge-pipeline Skill It is listed under DevOps & Infra on SkillMD.

Is Knowledge Pipeline safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Knowledge Pipeline?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Knowledge Pipeline free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Knowledge Pipeline?

S3YED (@s3yed) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Knowledge Pipeline by s3yed — Agent Skill

Name: Knowledge Pipeline
Author: S3YED

File contents

knowledge-pipeline Skill

Purpose

Operate the local-first Weblyfe knowledge pipeline: ingest documents and Google Drive / Notion content, embed locally, search by meaning, and surface it as a 3D graph in Mission Control. Built 2026-06-03. Codifies the patterns so future sessions and fleet agents do not re-derive them.

Where it lives

~/clawd/projects/knowledge-pipeline/. Store at .data/knowledge.db (SQLite, gitignored). Drive cache .data/drive-cache/, Notion cache .data/notion-cache/.

Core principle (from the adversarial review)

Local-first is REAL: SQLite + local embeddings are canonical. Pinecone is a deferred optional sync target, never the critical path. No client data is embedded until per-tenant isolation is hard. tenant is fail-closed (NOT NULL, scoped search never returns another tenant or UNASSIGNED).

Quick reference

Task	Command
Ingest a local folder	`python3 ingest.py <dir> --tenant weblyfe-internal`
Search	`python3 query.py "your question" --tenant weblyfe-internal -k 5`
Sync a Google Drive folder	`python3 sync_drive.py <driveFolderId>` then ingest `.data/drive-cache/<id>`
Export Notion (streaming)	`NOTION_API_KEY=... python3 -u sync_notion.py` then ingest `.data/notion-cache --tenant notion`

Stack

Embeddings: local Ollama bge-m3 (1024-dim). ollama pull bge-m3. Helper kp/embed.py hits localhost:11434/api/embed. Nothing leaves the machine.
Store: SQLite + brute-force numpy cosine (kp/store.py). Corpus is small (tens of thousands of chunks) so this is fast and avoids fragile native installs on Python 3.14. sqlite-vec / Chroma is the upgrade path.
Parse: PyMuPDF (fitz, preinstalled) for PDF, plain read for md/txt (kp/parse.py). iter_files skips node_modules/.git/.next/etc.
Incremental: files are skipped by md5 on re-run.

Gotchas

Python 3.14 (/opt/homebrew/bin/python3) has a broken pyexpat for native builds; chromadb/sentence-transformers installs are fragile. Stick to ollama + SQLite + numpy.
Notion /v1/search returns thousands of page objects for a large workspace. Stream (write per page) instead of collecting all first, or it appears to hang.
Notion rate limit is ~3 req/s; keep a 0.34s pause per request.
nohup buffers stdout; use python3 -u for live progress.

Mission Control integration

The 3D graph lives at MC Memory > Graph (reagraph WebGL, already installed). src/app/api/memory/graph/route.ts surfaces the pipeline DB as a knowledge cluster (GROUP BY path, cap 350 nodes). The Graph tab was gated !isLocal; unlocked in local mode in memory-browser-panel.tsx. Override the DB path with env KNOWLEDGE_DB_PATH.

Next phases (not built)

Per-tenant namespaces then client folders; FastAPI read-only for MC; audio (faster-whisper) + OCR ingest; entity/relation knowledge graph behind a flag; Pinecone remote sync; clustering/analysis; Obsidian export; voice "ask the KB".

---name: knowledge-pipelinedescription: knowledge-pipeline Skill---# knowledge-pipeline Skill## PurposeOperate the local-first Weblyfe knowledge pipeline: ingest documents and Google Drive / Notion content, embed locally, search by meaning, and surface it as a 3D graph in Mission Control. Built 2026-06-03. Codifies the patterns so future sessions and fleet agents do not re-derive them.## Where it lives`~/clawd/projects/knowledge-pipeline/`. Store at `.data/knowledge.db` (SQLite, gitignored). Drive cache `.data/drive-cache/`, Notion cache `.data/notion-cache/`.## Core principle (from the adversarial review)Local-first is REAL: SQLite + local embeddings are canonical. Pinecone is a deferred optional sync target, never the critical path. No client data is embedded until per-tenant isolation is hard. `tenant` is fail-closed (NOT NULL, scoped search never returns another tenant or UNASSIGNED).## Quick reference| Task | Command ||------|---------|| Ingest a local folder | `python3 ingest.py <dir> --tenant weblyfe-internal` || Search | `python3 query.py "your question" --tenant weblyfe-internal -k 5` || Sync a Google Drive folder | `python3 sync_drive.py <driveFolderId>` then ingest `.data/drive-cache/<id>` || Export Notion (streaming) | `NOTION_API_KEY=... python3 -u sync_notion.py` then ingest `.data/notion-cache --tenant notion` |## Stack- Embeddings: local Ollama `bge-m3` (1024-dim). `ollama pull bge-m3`. Helper `kp/embed.py` hits `localhost:11434/api/embed`. Nothing leaves the machine.- Store: SQLite + brute-force numpy cosine (`kp/store.py`). Corpus is small (tens of thousands of chunks) so this is fast and avoids fragile native installs on Python 3.14. sqlite-vec / Chroma is the upgrade path.- Parse: PyMuPDF (fitz, preinstalled) for PDF, plain read for md/txt (`kp/parse.py`). `iter_files` skips node_modules/.git/.next/etc.- Incremental: files are skipped by md5 on re-run.## Gotchas- Python 3.14 (`/opt/homebrew/bin/python3`) has a broken pyexpat for native builds; chromadb/sentence-transformers installs are fragile. Stick to ollama + SQLite + numpy.- Notion `/v1/search` returns thousands of page objects for a large workspace. Stream (write per page) instead of collecting all first, or it appears to hang.- Notion rate limit is ~3 req/s; keep a 0.34s pause per request.- nohup buffers stdout; use `python3 -u` for live progress.## Mission Control integrationThe 3D graph lives at MC `Memory > Graph` (reagraph WebGL, already installed). `src/app/api/memory/graph/route.ts` surfaces the pipeline DB as a `knowledge` cluster (GROUP BY path, cap 350 nodes). The Graph tab was gated `!isLocal`; unlocked in local mode in `memory-browser-panel.tsx`. Override the DB path with env `KNOWLEDGE_DB_PATH`.## Next phases (not built)Per-tenant namespaces then client folders; FastAPI read-only for MC; audio (faster-whisper) + OCR ingest; entity/relation knowledge graph behind a flag; Pinecone remote sync; clustering/analysis; Obsidian export; voice "ask the KB".

Frequently asked questions

Knowledge Pipeline