# Data Pipelines

> Data Pipelines (Router)

- Skill: `methasit-pun/data-pipelines` (Agent Skill)
- Install (CLI): `npx skillmds@latest add methasit-pun/data-pipelines`
- Raw SKILL.md: https://api.skillmd.com/api/skills/methasit-pun/data-pipelines/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Data & Analytics
- Author: Methasit-Pun (https://skillmd.com/u/methasit-pun)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/methasit-pun/data-pipelines

---


# Data Pipelines (Router)

This is a **router skill**. It groups the four skills that deal with getting data from a source into a destination and keeping it flowing. Diagnose which sub-area(s) the task touches, then **invoke the matching sub-skill(s) with the Skill tool**. For a task that spans areas, invoke several and combine their guidance.

## How to route

| If the task is about… | Invoke sub-skill |
|---|---|
| Designing a new ingestion job end-to-end: extraction strategy, idempotency, load patterns, raw/staging/marts layers, ELT vs ETL | `pipeline-design` |
| Scheduling multi-step pipelines, task dependencies, retries, SLAs, sensors, failure recovery, choosing between Airflow/Prefect/Dagster | `orchestration-patterns` |
| Real-time / near-real-time processing, Kafka/Flink/Kinesis/Spark Structured Streaming, consumer lag, windowing, exactly-once, late data | `streaming-patterns` |
| Moving between systems safely: cutover planning, backfill, dual-write, shadow reads, validation, rollback, retiring a legacy pipeline | `data-migration` |

## Routing rules

- **Default new-pipeline questions to `pipeline-design` first** — it frames source shape, volume, change pattern, and downstream contract, which the others build on.
- **Batch vs. streaming undecided?** Invoke `streaming-patterns` (it contains the stream-vs-batch decision) before committing to an architecture.
- **Anything scheduled with >2 dependent steps** also pulls in `orchestration-patterns`.
- **Replacing or moving off an existing system** always pulls in `data-migration`, usually alongside `pipeline-design`.
- Invoke via the Skill tool by name, e.g. call `Skill(skill="orchestration-patterns")`. Combine outputs; don't paraphrase from memory.

## Related groups

- Modeling the data once it lands → [[data-modeling]]
- Validating and governing it → [[data-reliability]]
- Where it runs and what it costs → [[cloud-data-infra]]