Source Intake

Source Intake

the first stage of a market voice run. it takes whatever raw voice the operator brings and turns it into an indexed, traceable corpus the rest of the pipeline can stand on. nothing downstream is trustworthy if intake is sloppy, because every claim eventually resolves back to a file this stage puts in raw/.

one rule above all others: capture speaker_type and source_type at import time. reconstructing who spoke later is exactly the guessing the kit exists to eliminate. if the operator's data carries those columns, keep them. if it does not, ask or default with a confidence downgrade. never infer the speaker from the text after the fact.

When to use

use when the operator has raw market voice and needs it ingested and indexed before extraction. invocable on its own (operator just wants a clean source index) or as Stage 1 of the frontrun orchestrator.

do not use to extract evidence cards or write the voice map: that is voice-extractor.

Inputs

-> a CSV following the canonical input contract, or a folder of .md/.txt files (one source per file or clearly delimited). -> the confirmed run base path and <market-slug> from the orchestrator's Stage 0, or ask the operator if running standalone. -> the scope facts from intake: market, business type, competitors, audience, goal.

Canonical input contract

the only hard-required column is verbatim. everything else degrades gracefully with defaults and a confidence note.

source_id (auto)  ·  verbatim (REQUIRED)  ·  speaker_type (rec)  ·  source_type (rec)
·  platform  ·  source_url  ·  date  ·  rating  ·  speaker_label
·  is_public (public|private|anonymized)  ·  notes

speaker_type taxonomy and source_type taxonomy live in references/source-types.md. read it before mapping columns. what makes a row a usable receipt (and when to downgrade confidence) lives in references/receipt-quality-rules.md.

Procedure

1. Confirm the run folder

confirm the base path. write nothing until it is confirmed. create:

frontrun-runs/YYYY-MM-DD-<market-slug>/
  raw/

2. Normalize

run the normalizer on the operator's input:

python scripts/normalize_sources.py <input-csv-or-folder> --out frontrun-runs/YYYY-MM-DD-<market-slug>/

it produces a canonical CSV and the populated raw/ folder, and seeds 01-source-index.md. if the script is not yet present in this install, do the normalization by hand against the canonical contract: assign src_001 ascending, write one excerpt file per source into raw/ named <source>-<type>-<id>.md, and build the index manually. either way the output shape is identical.

map the operator's columns to the canonical contract. when a recommended column is missing, apply the default from source-types.md and record the downgrade in 00-intake.md.

3. Populate raw/

every source gets an unpolished excerpt file in raw/. this forces the market's actual language to stay visible and gives the traceability chain its anchor. naming: <source>-<type>-<id>.md or the imported file's own name. one source file may later yield multiple evidence cards, all pointing back to its src_id.

4. Write 00-intake.md

scope: market/category, business type, sources in and out of scope, competitors, audience, confidence limits, and what this run should produce. if a public hard number is being claimed (e.g. total conversation count), the exact traceable figure lives here, nowhere more public.

5. Write 01-source-index.md

one row per source: src_id, source_type, platform, url/path, date, reliability (from the receipt-quality rules), and is_public as public|private|anonymized. this index is the middle link of the traceability chain: evidence_id -> card -> source_id -> this row -> file in raw/. a row with no raw/ file is a broken chain and must not exist.

6. Compute corpus sufficiency and write 00b-scope-contract.md

the corpus-sufficiency gate. record:

-> source_count (total indexed sources) -> distinct_source_types (how many of the taxonomy are represented) -> recency window (oldest to newest date) -> thin_pool (boolean) -> corpus_limitation_note (required whenever thin_pool == true, plain language)

thin_pool is true when source_count is below threshold OR distinct_source_types is below threshold. canonical thresholds (the same numbers validate_run.py enforces): fewer than 12 sources, or fewer than 3 distinct source types. when thin, say so plainly in the note. the orchestrator caps the angle slate at 3 and the validator enforces it. never launder thin comment noise into survey math (no "73% of customers" from 40 comments).

Outputs

References

• references/source-types.md: the source_type and speaker_type taxonomies and how each maps to reliability.
• references/receipt-quality-rules.md: what makes a usable receipt, when to downgrade confidence, and is_public handling.