REST API Pipeline Workflow

New ingestion pipeline

Workflow Entry

ALWAYS start with Find source (find-source) SKILL — discover the right dlt source for the user's data provider

Core workflow

1. Create pipeline (create-rest-api-pipeline) — scaffold, write code, configure credentials
2. Debug pipeline (debug-pipeline) — run it, inspect traces and load packages, fix errors
3. Validate data (validate-data) — inspect schema and data, fix types and structures, iterate until user is satisfied

Extend and harden

4. Deploy to dltHub Platform — hand off to dlthub-platform to deploy and run the pipeline on dltHub; can be done with a working pipeline
5. Adjust endpoint (adjust-endpoint) — add pagination, remove limits, add hints, mappings, correct schema etc.
6. Add incremental loading — set up dlt.sources.incremental, merge keys, and lag windows for production efficiency
7. Add endpoints (new-endpoint) — add more resources to the source
8. View data (view-data) — show data to the user & query and explore loaded data in Python

Handover to other toolkits

Incoming (to rest-api-pipeline)

• From dlthub-platform (from deploy-workspace when the pipeline needs modification before deploying) — pipeline name and destination are already known; skip find-source discovery and go straight to the relevant fix skill (debug-pipeline, adjust-endpoint, or new-endpoint).
• From quick-start (after path confirmation in quick-start) — the source name is passed as find-source's first argument. find-source should treat it as the discovery seed and skip the "what data do you want to extract?" question. The chosen path name (Discover / Inspect / Production / Full CDM) is informational only and does not change find-source's behaviour; downstream toolkit handoffs follow this toolkit's normal Outgoing rules.

Outgoing (from rest-api-pipeline)

When the user's needs go beyond this toolkit, hand over to:

• data-exploration — after validate-data or view-data, when the user wants interactive notebooks, charts, dashboards, or deeper analysis with marimo
• transformations — after validate-data or view-data, when the user wants to model the ingested data into a CDM or run cross-source transformations
• data-quality — after validate-data, when the user wants ongoing validation, check contracts, or quality guarantees on every pipeline load
• dlthub-platform — two entry points:
  • Early (after create-rest-api-pipeline or debug-pipeline): when the user wants to run the pipeline on dltHub right away — a working pipeline is enough to deploy
  • Later (after adjust-endpoint, incremental loading, add-endpoints, or a subsequent debug-pipeline run): when the pipeline is refined and the user wants to deploy or schedule it on dltHub
• filesystem-pipeline — from (find-source) when the user's data source is file-based (S3, GCS, local CSV, SFTP, etc.) rather than a REST API

Source: ota2000/dlt-community-sources — distributed by TomeVault.