# Source Intake > Ingests the operator's raw market voice (a CSV or a folder of .md/.txt files), normalizes it to the canonical input contract, indexes every source, and computes corpus sufficiency. Captures speaker_type and source_type at import so they are never guessed later. Triggers on: source intake, ingest my reviews, normalize sources, index my market voice, import my CSV of comments, build the source index, check if my corpus is thin. - Skill: `themattberman/source-intake` (Agent Skill, multi-file: 3 files) - Install (CLI): `npx skillmds@latest add themattberman/source-intake` - Raw SKILL.md: https://api.skillmd.com/api/skills/themattberman/source-intake/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Data & Analytics - Author: TheMattBerman (https://skillmd.com/u/themattberman) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/themattberman/source-intake --- # Source Intake the first stage of a market voice run. it takes whatever raw voice the operator brings and turns it into an indexed, traceable corpus the rest of the pipeline can stand on. nothing downstream is trustworthy if intake is sloppy, because every claim eventually resolves back to a file this stage puts in `raw/`. one rule above all others: capture `speaker_type` and `source_type` at import time. reconstructing who spoke later is exactly the guessing the kit exists to eliminate. if the operator's data carries those columns, keep them. if it does not, ask or default with a confidence downgrade. never infer the speaker from the text after the fact. ## When to use use when the operator has raw market voice and needs it ingested and indexed before extraction. invocable on its own (operator just wants a clean source index) or as Stage 1 of the `frontrun` orchestrator. do not use to extract evidence cards or write the voice map: that is `voice-extractor`. ## Inputs -> a CSV following the canonical input contract, or a folder of `.md`/`.txt` files (one source per file or clearly delimited). -> the confirmed run base path and `` from the orchestrator's Stage 0, or ask the operator if running standalone. -> the scope facts from intake: market, business type, competitors, audience, goal. ### Canonical input contract the only hard-required column is `verbatim`. everything else degrades gracefully with defaults and a confidence note. ``` source_id (auto) · verbatim (REQUIRED) · speaker_type (rec) · source_type (rec) · platform · source_url · date · rating · speaker_label · is_public (public|private|anonymized) · notes ``` `speaker_type` taxonomy and `source_type` taxonomy live in `references/source-types.md`. read it before mapping columns. what makes a row a usable receipt (and when to downgrade confidence) lives in `references/receipt-quality-rules.md`. ## Procedure ### 1. Confirm the run folder confirm the base path. write nothing until it is confirmed. create: ``` frontrun-runs/YYYY-MM-DD-/ raw/ ``` ### 2. Normalize run the normalizer on the operator's input: ```bash python scripts/normalize_sources.py --out frontrun-runs/YYYY-MM-DD-/ ``` it produces a canonical CSV and the populated `raw/` folder, and seeds `01-source-index.md`. if the script is not yet present in this install, do the normalization by hand against the canonical contract: assign `src_001` ascending, write one excerpt file per source into `raw/` named `--.md`, and build the index manually. either way the output shape is identical. map the operator's columns to the canonical contract. when a recommended column is missing, apply the default from `source-types.md` and record the downgrade in `00-intake.md`. ### 3. Populate raw/ every source gets an unpolished excerpt file in `raw/`. this forces the market's actual language to stay visible and gives the traceability chain its anchor. naming: `--.md` or the imported file's own name. one source file may later yield multiple evidence cards, all pointing back to its `src_id`. ### 4. Write 00-intake.md scope: market/category, business type, sources in and out of scope, competitors, audience, confidence limits, and what this run should produce. if a public hard number is being claimed (e.g. total conversation count), the exact traceable figure lives here, nowhere more public. ### 5. Write 01-source-index.md one row per source: `src_id`, `source_type`, `platform`, `url`/`path`, `date`, `reliability` (from the receipt-quality rules), and `is_public` as `public|private|anonymized`. this index is the middle link of the traceability chain: `evidence_id` -> card -> `source_id` -> this row -> file in `raw/`. a row with no `raw/` file is a broken chain and must not exist. ### 6. Compute corpus sufficiency and write 00b-scope-contract.md the corpus-sufficiency gate. record: -> `source_count` (total indexed sources) -> `distinct_source_types` (how many of the taxonomy are represented) -> recency window (oldest to newest `date`) -> `thin_pool` (boolean) -> `corpus_limitation_note` (required whenever `thin_pool == true`, plain language) `thin_pool` is true when `source_count` is below threshold OR `distinct_source_types` is below threshold. canonical thresholds (the same numbers `validate_run.py` enforces): fewer than 12 sources, or fewer than 3 distinct source types. when thin, say so plainly in the note. the orchestrator caps the angle slate at 3 and the validator enforces it. never launder thin comment noise into survey math (no "73% of customers" from 40 comments). ## Outputs | File | What it carries | |---|---| | `00-intake.md` | scope, business type, sources in/out, competitors, audience, confidence limits, goal | | `00b-scope-contract.md` | `source_count`, `distinct_source_types`, recency window, `thin_pool`, `corpus_limitation_note` | | `01-source-index.md` | every source: `src_id`, `source_type`, platform, url/path, date, reliability, is_public | | `raw/` | one unpolished excerpt file per source. the anchor of every traceability chain. | ## References - `references/source-types.md`: the `source_type` and `speaker_type` taxonomies and how each maps to reliability. - `references/receipt-quality-rules.md`: what makes a usable receipt, when to downgrade confidence, and `is_public` handling.