Methasit-Pun Data Pipelines (Router)
This is a router skill. It groups the four skills that deal with getting data from a source into a destination and keeping it flowing. Diagnose which sub-area(s) the task touches, then invoke the matching sub-skill(s) with the Skill tool. For a task that spans areas, invoke several and combine their guidance.
How to route
| If the task is about… | Invoke sub-skill |
|---|---|
| Designing a new ingestion job end-to-end: extraction strategy, idempotency, load patterns, raw/staging/marts layers, ELT vs ETL | pipeline-design |
| Scheduling multi-step pipelines, task dependencies, retries, SLAs, sensors, failure recovery, choosing between Airflow/Prefect/Dagster | orchestration-patterns |
| Real-time / near-real-time processing, Kafka/Flink/Kinesis/Spark Structured Streaming, consumer lag, windowing, exactly-once, late data | streaming-patterns |
| Moving between systems safely: cutover planning, backfill, dual-write, shadow reads, validation, rollback, retiring a legacy pipeline | data-migration |
Routing rules
- Default new-pipeline questions to
pipeline-designfirst — it frames source shape, volume, change pattern, and downstream contract, which the others build on. - Batch vs. streaming undecided? Invoke
streaming-patterns(it contains the stream-vs-batch decision) before committing to an architecture. - Anything scheduled with >2 dependent steps also pulls in
orchestration-patterns. - Replacing or moving off an existing system always pulls in
data-migration, usually alongsidepipeline-design. - Invoke via the Skill tool by name, e.g. call
Skill(skill="orchestration-patterns"). Combine outputs; don't paraphrase from memory.
Related groups
- Modeling the data once it lands → [[data-modeling]]
- Validating and governing it → [[data-reliability]]
- Where it runs and what it costs → [[cloud-data-infra]]