# Local Ingest > Onboard or backfill a company's changelog locally in Claude Code by fetching pages and extracting releases with the agent itself (and parallel sub-agents), then writing through the batch-upsert endpoint — no remote fetch dispatch, no server-side extraction inference billing. Use when you want to onboard a source without triggering the remote update workflow's billed extraction, or recover from a remote fetch that burned tokens and wrote zero releases. Includes a mandatory robots.txt / Content-Signal opt-out gate. - Skill: `buildinternet/local-ingest` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add buildinternet/local-ingest` - Raw SKILL.md: https://api.skillmd.com/api/skills/buildinternet/local-ingest/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: buildinternet (https://skillmd.com/u/buildinternet) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/buildinternet/local-ingest --- # Local Ingest Onboard a changelog **without dispatching the remote fetch path.** The remote path (`releases admin source fetch` → `POST /v1/workflows/update` → the API worker's `DeterministicUpdateWorkflow`, #1946) runs the body→records extraction server-side (incremental Haiku or the tool-loop) and bills that inference — even when it writes zero releases. A local Claude Code agent is already a capable model with web-fetch + file tools, so it can do the per-page extraction inline (folded into the session you're already in) and only needs a deterministic write path. **Core principle: the agent is the extractor.** You fetch, you parse, you structure the records, and you POST them to the batch endpoint, which does a plain idempotent upsert with **no AI on insert**. The remote extraction loop never runs. **Local Claude Code only.** This skill uses the `Agent` tool for sub-agent fan-out and assumes a persistent local filesystem and the CLI's `RELEASES_API_*` env. It sits alongside `seeding-playbooks` as a local-only operator skill — it is **not** deployed to the managed-agent fleet. ## Cost contract State this explicitly to the operator before a large run, and honor it: - **Spends:** your agent's own tokens for fetch + extract, plus any sub-agents you fan out (your choice of model — `sonnet` for quality, `haiku` for bulk). - **Does NOT spend:** no server-side extraction (the update workflow's incremental-Haiku / tool-loop passes), no remote `web_fetch` extraction. `POST /v1/workflows/update` is **never called**. - **The batch endpoint runs no AI on insert** — no summarization, no marketing classifier, no feed enrichment. It does an `insert` plus a fire-and-forget vector embed (`waitUntil`). The AI fields (`title_generated` / `title_short` / `summary`) are intentionally left empty; see _Manual enrichment_ below for the deliberate follow-up paths. The negative signal that confirms no remote run happened: in Axiom `releases-cloudflare-logs` there is **no** `POST .../fetch?sessionId=det-…` and **no** `extract-deps-worker` event for the source. ## When to use - Onboarding a new source (or backfilling an existing one) interactively in Claude Code and you want to skip the remote extraction cost. - A remote fetch burned a full extraction loop and wrote **0 releases** (the Conductor / large-body / `max_tokens` failure mode) — extracting inline sidesteps the loop entirely. - The page is fetchable by your own web tools (the refusal below is a policy gate, not a fetch limit). **When NOT to use:** - **You're in a managed-agent session.** This is local-only (Agent-tool fan-out + local filesystem + CLI env). MA sessions should keep using the normal fetch path. - **The source has a clean, content-rich feed.** Just add it (`managing-sources`) and let the cron pipeline fetch it. Local-ingest is for cases where the agent doing the extraction inline is the right tool, not a replacement for feed/GitHub adapters. - **The publisher opts out** (preflight refuses — see Step 1). ## Workflow ### Step 1 — Preflight (mandatory safety gate) Run the opt-out check before fetching or writing **anything**: ```bash bun .claude/skills/local-ingest/preflight.ts --json ``` It fetches `robots.txt` (following apex→www redirects), parses the Cloudflare **Content Signals** policy, and gates on exit code: | Exit | Verdict | Action | | ---- | --------- | -------------------------------------------------------------------------------------------------------------- | | `0` | `proceed` | Permissive or absent — continue. | | `1` | `refuse` | `ai-input=no` declared. **STOP. Surface to the user.** Do not fetch or write. | | `2` | `unknown` | Couldn't fetch/parse robots.txt. Retry once; if still unknown, **surface to the user** — don't assume proceed. | **The gate is on `ai-input` only.** Local ingest is search/input ingestion — feeding a search index — which is exactly what `ai-input` governs. `ai-train=no` opts out of model *training*, a use this path doesn't perform, so `ai-train=no` alone (e.g. `vercel.com`) correctly **proceeds**. **This is the Conductor lesson.** `conductor.build` serves `Content-Signal: ai-train=no, search=yes, ai-input=no` — its org/source already sits paused in the registry for exactly this reason (`releases admin playbook conductor`). A local onboard must not silently spend tokens or ingest content against a publisher opt-out. **`conductor.build` must be refused** (via its `ai-input=no`) — it is the regression target for this gate. The refusal is a **policy** choice, not a technical limitation: `web_fetch` and CF Browser Rendering can still retrieve these pages (only Cloudflare's `/crawl` endpoint hard-enforced the signal for Conductor). Honor the opt-out anyway. Override **only** with explicit publisher permission documented by the operator — there is no silent bypass. The preflight also prints any `Sitemap:` URLs from robots.txt — keep them for Step 3. ### Step 2 — Resolve target Ensure the org and source exist before writing — releases attach to a `source_id`. Defer to **`managing-sources`** for create/naming/primary rules and to **`finding-changelogs`** for whether a structured feed exists (if it does, prefer the normal pipeline). Then: - Read the org playbook for parse hints: `releases admin playbook ` (see `managing-sources` → Playbooks). - Capture the **`sourceSlug`** and the **`src_…` id** from the created/resolved row — you need one of them for the batch URL. If the source is brand-new, create it (e.g. `releases admin source create "" --url --org [--primary]`) but **do not trigger a fetch** — that would dispatch the remote update workflow you're trying to avoid. ### Step 3 — Map pages Classify the page shape (`releases admin discovery evaluate --json` reports `pageStructure: single-page | index | unknown`): - **Single-page changelog** → one fetch in Step 4. - **Index → detail** → enumerate the per-release URLs. Prefer `/sitemap.xml` (the preflight surfaced it) filtered to the changelog path; otherwise parse the index HTML for per-release links. - **Client-rendered index** → the index is a JS shell whose cards load via a client-side fetch, so `curl`/non-JS scrape returns **zero** entries. Enumerate by rendering: drive the index with Claude in Chrome (or fan out a rendering sub-agent), paginating if the list is windowed, and collect the per-release links. **Then check whether the _detail_ pages are statically rendered** — they often are even when the index isn't — and if so bulk-`curl` those directly (HTML→markdown locally, no render, no AI bill). Harvey's `help.harvey.ai/release-notes` is exactly this: client-rendered index, static detail pages. Don't conclude "no releases" from an empty non-JS scrape of the index. Cap to a recent window (default **~25–50** most-recent releases) and **log what you skipped** — never silently truncate. Re-runs are idempotent, so a deeper backfill later just adds rows. ### Step 4 — Fetch + extract Reuse the **`parsing-changelogs`** conventions for type/version/date/media (see _Parity_ below). - **A few pages:** fetch and extract inline in this agent. - **Many pages:** fan out parallel sub-agents — one URL-slice each, each returning records in the batch schema. Use the `Agent`-tool dispatch pattern from **`seeding-playbooks`** (and **superpowers:dispatching-parallel-agents** for the fan-out discipline). Do **not** fetch release URLs in the parent agent — delegate to keep the parent's context clean. - **Parent-saves caveat:** sub-agents may be blocked from making the batch write (permission limits). Plan for it — have each sub-agent **return** its records in its result, and let the parent (you) do the writes in Step 5. Same pattern `seeding-playbooks` uses for note-saving. - **Extract media — don't ship text-only.** Pull the images/video out of each release body and populate `media`, unwrapping `_next/image` / `_vercel/image` optimizer wrappers to the underlying CDN URL (`normalizeMediaUrl` from `packages/rendering/src/media-url.ts`). A text-only first pass is a common miss caught only on operator review (Harvey shipped text-only, then had to re-extract `_next/image` → `cdn.sanity.io` and re-seed); the enrichment re-POST is `mode: "upsert-content"` (Step 5). Each record must match the batch schema (see Step 5) and the parity rules. **Optional — deterministic extraction (Approach A, monorepo only):** instead of extracting by hand you can call the production extract libs in `packages/adapters/src/extract/` — `extractFromBody`, `extractWithTools`, `runDirectFetchExtraction`. `ExtractDeps.cloudflare` is nullable and `repo` can be a noop, so they run with just an Anthropic key (proof: `scripts/smoke-toolloop.ts`). This trades agent tokens for direct Anthropic-API tokens but reproduces the production tool-loop exactly. These packages are `private: true` — reachable from the monorepo only, never the thin CLI. The default remains Approach B (the agent is the extractor). ### Step 5 — Write (batch upsert) POST records to the batch endpoint in chunks (~50/request keeps requests bounded; the handler does the D1 bind-limit chunking server-side): ```bash # Use the org-scoped path. The bare /v1/sources/:slug form accepts ONLY a typed # `src_…` id — a human slug there returns 400 bare_slug_rejected (slugs are # org-scoped, #690). With a typed id the bare form also works: # POST $RELEASES_API_URL/v1/orgs//sources//releases/batch # POST $RELEASES_API_URL/v1/sources//releases/batch curl -sS -X POST "$RELEASES_API_URL/v1/orgs//sources//releases/batch" \ -H "Authorization: Bearer $RELEASES_API_KEY" \ -H "Content-Type: application/json" \ -d @chunk.json # { "releases": [ … ] } ``` Per-release fields (the real accepted shape — `workers/api/src/routes/sources.ts`, `postReleasesBatchHandler`): ```jsonc { "version": "1.4.0", // optional; omit/null for date-headed or blog sources "title": "Dark mode", // required "content": "Markdown body…", // required "url": "https://…/changelog/dark-mode", // optional but ALWAYS populate — it's the dedup key "contentHash": "…", // optional "publishedAt": "2026-05-30", // optional; ISO-8601 (see Parity) "media": "https://…/shot.png", // optional "type": "feature", // optional; "feature" | "rollup" "prerelease": false, // optional } ``` - **Idempotent.** Upserts on `UNIQUE(source_id, url)`, **fill-don't-clobber**: on collision it only backfills content/media when the stored row is _empty_, so a re-run never overwrites a good row. Safe to re-run and to page — same `url` updates rather than duplicates. - **Enrichment re-POST (`mode: "upsert-content"`).** Because the default is fill-only, re-POSTing **richer content for a URL that already has content** is a no-op — the stub stays. For a deliberate second pass (seed index summaries first, then full detail-page bodies + media), add `"mode": "upsert-content"` at the top level of the batch body: it **overwrites** content/media on a same-URL collision and skips the scrape title-dedup pre-filter (#1526). Use it only for intentional enrichment; the default path stays fill-only so a routine re-fetch can't clobber. **The `url` must be identical to the seeded row** — enrich keys on URL, so a changed URL scheme inserts a duplicate instead. If the URL scheme changed, re-seed instead: `releases release delete --source --hard` then a fresh `/batch` (default mode). - **Auth.** Requires `write` scope; the static `RELEASES_API_KEY` (root) satisfies it. `.env` autoloads `RELEASES_API_URL` + `RELEASES_API_KEY` in this repo. - **No AI on insert** (see Cost contract). Chunking helper (reads a `records.json` array, posts in batches of 50): ```bash bun -e ' const orgSlug = process.argv[1], sourceSlug = process.argv[2], file = process.argv[3]; const all = JSON.parse(await Bun.file(file).text()); const base = process.env.RELEASES_API_URL, key = process.env.RELEASES_API_KEY; for (let i = 0; i < all.length; i += 50) { const releases = all.slice(i, i + 50); const res = await fetch(`${base}/v1/orgs/${orgSlug}/sources/${sourceSlug}/releases/batch`, { method: "POST", headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/json" }, body: JSON.stringify({ releases }), }); console.log(`chunk ${i / 50}: ${res.status} ${(await res.text()).slice(0, 200)}`); } ' records.json ``` ### Step 6 — Validate ```bash releases tail --json # or: get_latest_releases (typed tool) ``` Confirm releases have non-empty **titles, dates, content, and media**. If a row is _empty_, fix the extraction and **re-run** — the default upsert backfills empty rows safely. If a row already has _thin-but-non-empty_ content (e.g. an index-summary stub you now want to replace with the full body), a plain re-run is a no-op — use the `mode: "upsert-content"` enrichment re-POST from Step 5. **Recovering a source, not onboarding one?** This step only confirms what got written — it says nothing about what didn't. Before telling anyone a source is "recovered" or "nothing is missing," see `backfilling-sources` → _Before declaring a source "clean" or "recovered"_ for the required live-page spot check. ### Step 7 — Playbook note Record how this source was ingested so future agents don't re-derive it. Via `manage_playbook` (action `update_notes`) or `releases admin playbook --notes-file -`. Note that the source was **locally ingested** (not on the cron path yet), its observed cadence, and any extraction quirks. Follow the `managing-sources` playbook-authoring rubric (imperative voice, keep-test, `### Fetch instructions` / `### Traps` / `### Coverage`). Don't restate metadata the header already carries. ## Parity requirements Local output must match production shape so the source can later be picked up by the normal pipeline with no cleanup: | Axis | Rule | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` | `feature` vs `rollup` per **`parsing-changelogs`** → _Classifying Rollups_. Leave unset/`feature` by default. | | `version` | Real version string, or omit. **Pre-clean `` / `n/a` placeholders** — the batch path does not strip them (the single-insert path does, server-side). | | `publishedAt` | ISO-8601. Approximate from month/quarter/year headings rather than omit (`parsing-changelogs` → _Dates_): month → first of month, etc. | | `media` | Unwrap `_next/image` / Vercel optimizer wrappers before storing — use `normalizeMediaUrl` from `packages/rendering/src/media-url.ts`. | | `url` | **Always populate.** Dedup relies on a stable per-release `url`; without it the upsert can't collapse re-runs. | ## Manual enrichment (out of scope for `/batch`, but available) The batch path intentionally omits `title_generated` / `title_short` / `summary` and org overviews. When an operator wants them, set them deliberately, case by case: - **Per-release override:** `PATCH /v1/releases/:id` accepts `summary`, `titleGenerated`, `titleShort`, `composition`, `version`, … and re-embeds when content/title/AI-fields change. - **At insert time (alternative to `/batch`):** the single-release insert `POST /v1/sources/:slug/releases` accepts `summary?` / `titleGenerated?` / `titleShort?` — use this path instead of `/batch` when you want enrichment set in the same write. It also strips ``/`n/a` versions server-side. - **Bulk AI enrichment after the fact:** `bun scripts/generate-release-content.ts --orgs=`. ## Common mistakes - **Skipping the preflight.** It's the first step for a reason — silently ingesting an opt-out source is the exact failure this skill exists to prevent. `conductor.build` must be refused. - **Silent truncation.** Capping the window is fine; not telling the user what you skipped is not. - **Missing `url`.** Breaks idempotency — re-runs duplicate instead of upserting. - **Triggering a fetch on the new source.** `releases admin source fetch` / `manage_source` action `fetch` starts the remote update workflow (billed server-side extraction) — the cost you're avoiding. Create the source, then write via `/batch`. - **Fetching detail pages in the parent agent during fan-out.** Delegate to sub-agents; keep the parent context clean. - **Using this for a clean feed source.** Add it and let cron fetch — don't hand-extract what the feed adapter handles for free. - **Running it in a managed-agent session.** Local Claude Code only. ## Related skills - **`finding-changelogs`** — is there a feed/GitHub source that makes local-ingest unnecessary? Page-structure evaluation. - **`managing-sources`** — create org/source, naming, primary, playbook authoring. - **`parsing-changelogs`** — the extraction conventions you must match (type, dates, rollups). - **`seeding-playbooks`** — the parallel `Agent`-tool dispatch + parent-saves pattern reused in Step 4. - **superpowers:dispatching-parallel-agents** — fan-out discipline for the many-pages case.