# REST API Pipeline Workflow > ALWAYS read and follow this skill before acting. New ingestion pipeline Use when this capability is needed. - Skill: `tomevault-io/rest-api-pipeline-workflow` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/rest-api-pipeline-workflow` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/rest-api-pipeline-workflow/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/rest-api-pipeline-workflow --- # New ingestion pipeline ## Workflow Entry **ALWAYS** start with **Find source** (`find-source`) SKILL — discover the right dlt source for the user's data provider ## Core workflow 1. **Create pipeline** (`create-rest-api-pipeline`) — scaffold, write code, configure credentials 2. **Debug pipeline** (`debug-pipeline`) — run it, inspect traces and load packages, fix errors 3. **Validate data** (`validate-data`) — inspect schema and data, fix types and structures, iterate until user is satisfied ## Extend and harden 4. **Deploy to dltHub Platform** — hand off to **dlthub-platform** to deploy and run the pipeline on dltHub; can be done with a working pipeline 5. **Adjust endpoint** (`adjust-endpoint`) — add pagination, remove limits, add hints, mappings, correct schema etc. 6. **Add incremental loading** — set up `dlt.sources.incremental`, merge keys, and lag windows for production efficiency 7. **Add endpoints** (`new-endpoint`) — add more resources to the source 8. **View data** (`view-data`) — show data to the user & query and explore loaded data in Python ## Handover to other toolkits ### Incoming (to rest-api-pipeline) - From **dlthub-platform** (from `deploy-workspace` when the pipeline needs modification before deploying) — pipeline name and destination are already known; skip `find-source` discovery and go straight to the relevant fix skill (`debug-pipeline`, `adjust-endpoint`, or `new-endpoint`). - From **quick-start** (after path confirmation in `quick-start`) — the source name is passed as `find-source`'s first argument. `find-source` should treat it as the discovery seed and skip the "what data do you want to extract?" question. The chosen path name (Discover / Inspect / Production / Full CDM) is informational only and does not change `find-source`'s behaviour; downstream toolkit handoffs follow this toolkit's normal `Outgoing` rules. ### Outgoing (from rest-api-pipeline) When the user's needs go beyond this toolkit, hand over to: - **data-exploration** — after `validate-data` or `view-data`, when the user wants interactive notebooks, charts, dashboards, or deeper analysis with marimo - **transformations** — after `validate-data` or `view-data`, when the user wants to model the ingested data into a CDM or run cross-source transformations - **data-quality** — after `validate-data`, when the user wants ongoing validation, check contracts, or quality guarantees on every pipeline load - **dlthub-platform** — two entry points: - **Early** (after `create-rest-api-pipeline` or `debug-pipeline`): when the user wants to run the pipeline on dltHub right away — a working pipeline is enough to deploy - **Later** (after `adjust-endpoint`, incremental loading, `add-endpoints`, or a subsequent `debug-pipeline` run): when the pipeline is refined and the user wants to deploy or schedule it on dltHub - **filesystem-pipeline** — from (`find-source`) when the user's data source is file-based (S3, GCS, local CSV, SFTP, etc.) rather than a REST API --- > Source: [ota2000/dlt-community-sources](https://github.com/ota2000/dlt-community-sources) — distributed by [TomeVault](https://tomevault.io).