# Ingestion And Connectors

> Create, sync, and diagnose Brain sources across all providers (manual, generic, clips, slack, granola, github) — health states, sync scheduling, and credential resolution. Use when adding a source, running or debugging a sync, or the user asks why a source is stale, erroring, or not importing.

- Skill: `builderio/ingestion-and-connectors` (Agent Skill)
- Install (CLI): `npx skillmds@latest add builderio/ingestion-and-connectors`
- Raw SKILL.md: https://api.skillmd.com/api/skills/builderio/ingestion-and-connectors/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: Builder.io (https://skillmd.com/u/builderio)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/builderio/ingestion-and-connectors

---


# Ingestion and Connectors

For Slack-specific rollout steps (channel allow-lists, pilot runs, scope
checklist), read `brain-runbook` instead — this skill covers the
provider-agnostic source lifecycle.

## Source Providers

`create-source` accepts exactly `manual`, `generic`, `clips`, `slack`,
`granola`, or `github` (`sourceProviderSchema` in `actions/_schemas.ts`), and
rejects anything else. There is no arbitrary/custom
provider string — a generic webhook-fed source uses `provider: "generic"`
with a `sourceKey` + minted `ingestToken`, not a made-up provider id.

```bash
pnpm --filter brain action create-source \
  --title "Support call transcripts" \
  --provider generic \
  --sourceKey support-calls \
  --visibility org
```

`create-source` returns `ingestToken` in the response **once** — it is stored
only as a SHA-256 hash (`ingestTokenHash`) on the source row, so it cannot be
retrieved again later. Surface it to the user immediately (e.g. for wiring
into an external system's webhook config) rather than assuming you can read
it back with `get-source`.

Manual sources are push/import-driven and private by default. Use
`import-markdown-files` for a bounded folder or batch of `.md` / `.markdown`
files; it preserves relative paths for deduplication, stores searchable text as
document captures, and reports each file's import or privacy outcome. Use the
standard sharing actions to change a source to organization visibility or grant
specific access. Do not use `sync-source` for a manual source.

## Blessed FAQ And Docs Publishers

Approved FAQs, docs, handbooks, and similar owned resources use the same
`generic` signed source contract. They are not a new provider and do not require
Brain to crawl the entire upstream system. The source owner publishes only the
records Brain is allowed to use:

```bash
pnpm --filter brain action create-source \
  --title "Blessed Agent-Native docs" \
  --provider generic \
  --sourceKey agent-native-docs \
  --policy '{"trustTier":"blessed","answerEligible":true,"authority":100,"freshnessWindowDays":null,"reviewRequired":false,"conflictBehavior":"prefer-higher-authority"}' \
  --visibility org
```

The source answer policy is code-enforced:

| Field                 | Effect                                                                              |
| --------------------- | ----------------------------------------------------------------------------------- |
| `trustTier`           | `blessed`, `standard`, or `untrusted`; cited answers rank higher trust first.       |
| `answerEligible`      | Excludes the source from `ask-brain` answers when false.                            |
| `authority`           | Ranks otherwise eligible sources from 0 to 100.                                     |
| `freshnessWindowDays` | Excludes source-backed results after the configured window; `null` disables expiry. |
| `reviewRequired`      | Raw captures cannot support answers, and company knowledge enters review.           |
| `conflictBehavior`    | Prefer higher authority, surface conflicts, or require review before raw support.   |

Legacy sources remain `standard`, answer-eligible, authority 50, with no
freshness expiry. Use `update-source --policy ...` to tighten an existing
source without recreating it.

## Source Health States

`get-brain-health` (`readBrainHealth` in `server/lib/brain-health.ts`) is the
single action to check before telling a user "your source is broken" or
"nothing has synced yet." Each source gets one deterministic `health` value:

| Health | Meaning |
| --- | --- |
| `error` | Source `status === "error"`, has `lastError`, or its latest sync run failed. |
| `paused` | Source `status` is `paused` or `archived`. |
| `needs_setup` | Slack source with no configured channel allow-list yet (`channelIds`/`channels`/`allowedChannels` all empty). |
| `needs_sync` | Auto-sync-eligible provider (slack/granola/github) that has never completed a sync (`lastSyncedAt` is null). |
| `stale` | Past its computed `nextSyncAt` by more than a 15-minute grace window. |
| `healthy` | None of the above. |

`get-brain-health` also reports `distillationQueue` counts (`pending`,
`failed`, `stale` — a `processing` row untouched for 15+ minutes counts as
stale), `proposals.pending`, `sources.attention` (every non-healthy source),
and ordered `setup.nextSteps`. Read this before diagnosing a "why isn't X in
Brain yet" question — it is cheaper and more reliable than inspecting
individual sources by hand.

## Running a Sync

- `sync-source --sourceId=<id>` runs one source's connector immediately
  (requires `editor` access on that source). Slack scans only its allow-listed
  channels; Granola polls accessible notes; GitHub imports approved
  repository issues/PRs.
- `sync-due-sources` sweeps every accessible source whose `nextSyncAt` has
  passed. Prefer this for "catch everything up" requests instead of listing
  sources and calling `sync-source` in a loop.
- Auto-sync only applies to `slack`, `granola`, and `github` sources, and only
  when the source's own config doesn't explicitly set `autoSync: false`
  (`sourceAutoSync` in `brain-health.ts`). `manual`, `generic`, and `clips`
  sources are push/import-driven, not polled — there's no `nextSyncAt` to
  wait on for those.

## Beyond the Source Actions

Source sync actions are convenience readers, not integration limits. For ad hoc
provider analysis, or a question that needs an endpoint, filter, or payload the
source actions do not model, call `provider-api-catalog` / `provider-api-docs`
first, then `provider-api-request` against the provider's real HTTP API. Use
`connectionId` for a specific shared grant and `accountId` for a specific OAuth
account.

## Credential Resolution Order

For connector work, use existing workspace integration grants when available;
do not duplicate provider tokens into Brain. Every provider-backed source
resolves its credential in this order (never skip ahead or ask for a duplicate
token if an earlier tier already has one):

1. Granted `workspace_connections` / `workspace_connection_grants` for
   `appId=brain` — a shared credential another app or Dispatch already
   connected and granted to Brain.
2. Backward-compatible Brain-local SQL credentials (legacy, pre-workspace-grant
   sources).
3. Registered vault secrets scoped to the same user/org/workspace.

Brain source credentials **do not** fall back to raw deploy-level environment
variables — `.env`/`.env.local` alone will not satisfy a source's credential
check. `list-connection-providers` reports per-provider readiness
(`connected`, `granted`, `needs_grant`, `not_connected`) plus credential
health, so check that before telling a user to paste a new token — if a
provider shows `needs_grant`, the fix is granting Brain access to the existing
connection, not creating a new one.

## Editing And Removing Sources

- `update-source` edits title, config, cursor, status, or answer policy on an
  existing source.
- `delete-source` is a hard delete — there's no soft-archive alternative
  exposed as an action; setting `status: "paused"` via `update-source` is the
  reversible way to stop a source without losing its captures.
- `list-sources --provider=<p> --includeArchived=<bool>` is scoped by
  `accessFilter` — sharing/visibility rules apply the same way they do for
  knowledge and proposals.

## Related Skills

- `brain` — distillation, retrieval, and publish-tier mechanics once a
  capture exists.
- `brain-runbook` — Slack rollout steps, distillation worker internals,
  scheduled sync cron wiring, demo/eval seeding, and the generic ingest
  webhook payload shape.
- `sharing` — the `accessFilter`/`assertAccess` pattern used throughout source
  and capture reads.