Data Pipeline Etl

📊 Data Pipeline / ETL

Group Skill — Orchestrates sub-skills for building data ingestion, transformation, or analytics pipelines. Activate by saying: "Build a data pipeline" or "ETL setup"

When to Use

• Building data ingestion, transformation, or analytics pipelines
• Need to process and transform data at scale
• Setting up data observability and visualization

Step-by-Step Workflow

Step 1: Source

Skill: data-engineer / data-quality-frameworks

• Analyze data sources
• Assess data quality
• Define extraction strategy

Step 2: Ingest

Skill: snowflake-development / bigquery-data-transfer-service

• Set up data loading
• Configure data connectors
• Schedule ingestion jobs

Step 3: Transform

Skill: dbt-transformation-patterns / dbt-bigquery

• Define data transformations
• Create dbt models
• Test transformations

Step 4: Clean

Skill: data-autocleaning / data-quality-frameworks

• Deduplicate records
• Handle missing values
• Validate data quality

Step 5: Store

Skill: database / polars / data-scientist

• Choose storage solution
• Optimize for query patterns
• Partition and index

Step 6: Monitor

Skill: monte-carlo-monitor-creation / monte-carlo-push-ingestion

• Set up data observability
• Monitor data freshness
• Alert on anomalies

Step 7: Visualize

Skill: matplotlib / seaborn / plotly

• Create charts and dashboards
• Build reports
• Share insights

Completion

Data pipeline is built with ingestion, transformation, and visualization. 📊

Sub-Skills Referenced

1. data-engineer
2. data-quality-frameworks
3. snowflake-development
4. bigquery-data-transfer-service
5. dbt-transformation-patterns
6. dbt-bigquery
7. data-autocleaning
8. database
9. polars
10. data-scientist
11. monte-carlo-monitor-creation
12. monte-carlo-push-ingestion
13. matplotlib
14. seaborn
15. plotly