# Data Pipeline Etl

> Group skill: Data pipeline/ETL — source analysis, ingestion, transformation, cleaning, storage, monitoring, and visualization.

- Skill: `may215/data-pipeline-etl` (Agent Skill)
- Install (CLI): `npx skillmds@latest add may215/data-pipeline-etl`
- Raw SKILL.md: https://api.skillmd.com/api/skills/may215/data-pipeline-etl/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: may215 (https://skillmd.com/u/may215)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/may215/data-pipeline-etl

---


# 📊 Data Pipeline / ETL

> **Group Skill** — Orchestrates sub-skills for building data ingestion, transformation, or analytics pipelines.
> Activate by saying: "Build a data pipeline" or "ETL setup"

## When to Use

- Building data ingestion, transformation, or analytics pipelines
- Need to process and transform data at scale
- Setting up data observability and visualization

## Step-by-Step Workflow

### Step 1: Source
**Skill**: `data-engineer` / `data-quality-frameworks`
- Analyze data sources
- Assess data quality
- Define extraction strategy

### Step 2: Ingest
**Skill**: `snowflake-development` / `bigquery-data-transfer-service`
- Set up data loading
- Configure data connectors
- Schedule ingestion jobs

### Step 3: Transform
**Skill**: `dbt-transformation-patterns` / `dbt-bigquery`
- Define data transformations
- Create dbt models
- Test transformations

### Step 4: Clean
**Skill**: `data-autocleaning` / `data-quality-frameworks`
- Deduplicate records
- Handle missing values
- Validate data quality

### Step 5: Store
**Skill**: `database` / `polars` / `data-scientist`
- Choose storage solution
- Optimize for query patterns
- Partition and index

### Step 6: Monitor
**Skill**: `monte-carlo-monitor-creation` / `monte-carlo-push-ingestion`
- Set up data observability
- Monitor data freshness
- Alert on anomalies

### Step 7: Visualize
**Skill**: `matplotlib` / `seaborn` / `plotly`
- Create charts and dashboards
- Build reports
- Share insights

## Completion

Data pipeline is built with ingestion, transformation, and visualization. 📊

## Sub-Skills Referenced

1. `data-engineer`
2. `data-quality-frameworks`
3. `snowflake-development`
4. `bigquery-data-transfer-service`
5. `dbt-transformation-patterns`
6. `dbt-bigquery`
7. `data-autocleaning`
8. `database`
9. `polars`
10. `data-scientist`
11. `monte-carlo-monitor-creation`
12. `monte-carlo-push-ingestion`
13. `matplotlib`
14. `seaborn`
15. `plotly`