Skill: Ingest New Content
Trigger
User provides new posts, writer content, URLs, or newsletter URLs to add to the system.
Process
For new user posts
- Read the .docx or text content
- Analyze for voice patterns, new mental models, topic evolution
- Update
data/brain/pmm_brain.json voice profile if needed
- Store in ChromaDB for gap map retrieval
For new reference writer content
- Read the .docx, URL, or text content
- Analyze for mental models, argumentation patterns, thinking process
- Update the PMM Brain's synthesized models if new patterns found
- Store in ChromaDB for gap map retrieval
For newsletter/blog URLs
- Fetch and read the content via WebFetch
- Extract writer identity, key themes, mental models, unique perspectives
- Add to
newsletter_insights section of the Brain
- Note any new emerging beliefs for the value landscape
For newsletter scraping (Substack / Beehiiv)
When user provides a newsletter URL for scraping (e.g., "scrape this newsletter", "ingest posts from https://..."):
Scrape — Run the newsletter scraper to pull posts:
python3 -m src.ingestion.newsletter_scraper \
--url "https://newsletter-url.com" \
--days 90 --full-content
This auto-detects the platform (Substack or Beehiiv), scrapes posts,
stores them in SQLite + ChromaDB, and saves raw JSON to data/scraped/.
Generate brain brief — Get the synthesis brief:
python3 -m src.ingestion.newsletter_scraper \
--url "https://newsletter-url.com" \
--days 90 --full-content --output brain-brief
Synthesize into Brain — Read the brain brief and update data/brain/pmm_brain.json:
newsletter_insights: Add/update entry with focus, key_frameworks, key_data_points, contrarian_insights, useful_forevidence_bank: Extract concrete stats, benchmarks, data pointspmm_value_landscape: Note contested or emerging beliefssynthesized_mental_models: Extract new mental models or frameworkstopic_depth_layers: Add practitioner wisdom, common mistakes, key debates
Verify — Confirm what was added to each Brain section.
CLI flags
| Flag |
Description |
--url |
Newsletter URL (required) |
--name |
Source name (auto-detected from author if omitted) |
--days N |
Only scrape last N days |
--max-posts N |
Limit total posts |
--keywords "a,b" |
Filter by keywords |
--full-content |
Fetch full post bodies (slower, richer for synthesis) |
--no-store |
Skip SQLite/ChromaDB storage, just preview |
--output |
summary (default), json, or brain-brief |
Supported platforms
- Substack — uses public
/api/v1/archive JSON endpoint
- Beehiiv — uses hidden
/posts JSON endpoint
- Both work with custom domains (auto-detected)
For visual illustrations
- Read images to understand visual style
- Update visual signature in voice profile if patterns have evolved
Data Flow
Newsletter URL
→ newsletter_scraper.py (detect platform → scrape → filter)
→ data/scraped/{source}_{timestamp}.json (audit trail)
→ SQLite posts table + ChromaDB embeddings (retrieval)
→ Brain synthesis brief (for Claude analysis)
→ pmm_brain.json updates (newsletter_insights, evidence_bank, etc.)
Output
Confirm what was ingested and what changed in the Brain.
1---2name: ingest3description: Skill: Ingest New Content4---5# Skill: Ingest New Content67## Trigger8User provides new posts, writer content, URLs, or newsletter URLs to add to the system.910## Process1112### For new user posts131. Read the .docx or text content142. Analyze for voice patterns, new mental models, topic evolution153. Update `data/brain/pmm_brain.json` voice profile if needed164. Store in ChromaDB for gap map retrieval1718### For new reference writer content191. Read the .docx, URL, or text content202. Analyze for mental models, argumentation patterns, thinking process213. Update the PMM Brain's synthesized models if new patterns found224. Store in ChromaDB for gap map retrieval2324### For newsletter/blog URLs251. Fetch and read the content via WebFetch262. Extract writer identity, key themes, mental models, unique perspectives273. Add to `newsletter_insights` section of the Brain284. Note any new emerging beliefs for the value landscape2930### For newsletter scraping (Substack / Beehiiv)3132When user provides a newsletter URL for scraping (e.g., "scrape this newsletter", "ingest posts from https://..."):33341. **Scrape** — Run the newsletter scraper to pull posts:35 ```bash36 python3 -m src.ingestion.newsletter_scraper \37 --url "https://newsletter-url.com" \38 --days 90 --full-content39 ```40 This auto-detects the platform (Substack or Beehiiv), scrapes posts,41 stores them in SQLite + ChromaDB, and saves raw JSON to `data/scraped/`.42432. **Generate brain brief** — Get the synthesis brief:44 ```bash45 python3 -m src.ingestion.newsletter_scraper \46 --url "https://newsletter-url.com" \47 --days 90 --full-content --output brain-brief48 ```49503. **Synthesize into Brain** — Read the brain brief and update `data/brain/pmm_brain.json`:51 - `newsletter_insights`: Add/update entry with focus, key_frameworks, key_data_points, contrarian_insights, useful_for52 - `evidence_bank`: Extract concrete stats, benchmarks, data points53 - `pmm_value_landscape`: Note contested or emerging beliefs54 - `synthesized_mental_models`: Extract new mental models or frameworks55 - `topic_depth_layers`: Add practitioner wisdom, common mistakes, key debates56574. **Verify** — Confirm what was added to each Brain section.5859#### CLI flags60| Flag | Description |61|------|-------------|62| `--url` | Newsletter URL (required) |63| `--name` | Source name (auto-detected from author if omitted) |64| `--days N` | Only scrape last N days |65| `--max-posts N` | Limit total posts |66| `--keywords "a,b"` | Filter by keywords |67| `--full-content` | Fetch full post bodies (slower, richer for synthesis) |68| `--no-store` | Skip SQLite/ChromaDB storage, just preview |69| `--output` | `summary` (default), `json`, or `brain-brief` |7071#### Supported platforms72- **Substack** — uses public `/api/v1/archive` JSON endpoint73- **Beehiiv** — uses hidden `/posts` JSON endpoint74- Both work with custom domains (auto-detected)7576### For visual illustrations771. Read images to understand visual style782. Update visual signature in voice profile if patterns have evolved7980## Data Flow8182```83Newsletter URL84 → newsletter_scraper.py (detect platform → scrape → filter)85 → data/scraped/{source}_{timestamp}.json (audit trail)86 → SQLite posts table + ChromaDB embeddings (retrieval)87 → Brain synthesis brief (for Claude analysis)88 → pmm_brain.json updates (newsletter_insights, evidence_bank, etc.)89```9091## Output92Confirm what was ingested and what changed in the Brain.
raven-sourav