Data Pipeline Engineering

Use this skill when designing, building, or debugging data pipelines — ETL/ELT workflows, data orchestration (Airflow/Dagster/Prefect), streaming (Kafka/Spark Streaming), schema drift handling, pipeline observability, backfill strategies, and data warehouse integration (BigQuery/Snowflake/Redshift). Covers batch and real-time patterns, idempotency, partitioning, incremental loads, and monitoring.

JPeetz Updated 16 repo stars

File contents

JPeetz/agent-skills commit 3f8557274d

Frequently asked questions

npx get-skill add jpeetz/data-pipeline-engineering