# Data Etl Pipeline

> Design ETL / ELT pipelines — extraction, transformation, load — with reproducible steps, schema evolution, quality checks, and observability.

- Skill: `quantumquirkxyz/data-etl-pipeline` (Agent Skill)
- Install (CLI): `npx skillmds@latest add quantumquirkxyz/data-etl-pipeline`
- Raw SKILL.md: https://api.skillmd.com/api/skills/quantumquirkxyz/data-etl-pipeline/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: quantumquirkxyz (https://skillmd.com/u/quantumquirkxyz)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/quantumquirkxyz/data-etl-pipeline

---


## Contract

- **Input:** source data descriptions, target schema, quality requirements.
- **Output:** pipeline architecture + quality rules + schema evolution.
- **Side effects:** may process data when executed (read-only or write to target).
- **Dependencies:** source access, target access.
- **Stop condition:** architecture saved; quality rules defined.
- **Risk:** medium — data corruption risks; requires testing.
- **Boundary:** defines pipeline; execution requires approval.

# ETL Pipeline Design

Design an **ETL / ELT pipeline** — extraction, transformation, load — with reproducible steps, schema evolution, and data quality checks.

## Process

### 1. Source analysis
- Source type: database (SQL / NoSQL), file (CSV / JSON / Parquet / ORC), API (REST / GraphQL / gRPC), stream (Kafka / Kinesis / Pub/Sub).
- Schema: columns, types, constraints.
- Frequency: batch (hourly / daily) or streaming (near real-time).
- Volume: rows / GB per run.
- Quality issues: missing values, out-of-range, duplicates, format errors.

**Completion criterion:** source profile saved.

### 2. Transformation design
- Filter: remove rows that don't meet criteria.
- Clean: fix types, trim whitespace, standardise formats.
- Aggregate: group by key, compute statistics.
- Enrich: join with reference data (customer, product, geography).
- Deduplicate: exact / fuzzy / probabilistic.

**Completion criterion:** transformation steps documented.

### 3. Load design
- Target type: warehouse (Snowflake / BigQuery / Redshift / Databricks) vs database vs lake.
- Load strategy: full refresh (replace all) / incremental (append new / update changed) / merge (upsert).
- Partition / clustering strategy for performance.
- Backfill plan for historical data.

**Completion criterion:** load strategy saved.

### 4. Data quality
- Null rate per column (threshold: < X%).
- Out-of-range checks (min / max / expected range).
- Duplicate rate.
- Referential integrity (foreign key consistency).
- Freshness (data delay from source to target).

**Completion criterion:** quality rules saved with thresholds and actions (fail / warn / fix).

### 5. Schema evolution
- How to handle new columns (add, ignore, raise error).
- How to handle renamed / deleted columns.
- Versioning: schema registry (e.g. Confluent Schema Registry, AWS Glue Data Catalog).

**Completion criterion:** evolution plan saved.