How do I install the Data Etl Pipeline skill?

Run npx skillmds@latest add quantumquirkxyz/data-etl-pipeline in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Data Etl Pipeline safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Data Etl Pipeline?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Data Etl Pipeline free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Data Etl Pipeline?

quantumquirkxyz (@quantumquirkxyz) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Data Etl Pipeline by quantumquirkxyz

Name: Data Etl Pipeline
Author: quantumquirkxyz

File contents

Contract

Input: source data descriptions, target schema, quality requirements.
Output: pipeline architecture + quality rules + schema evolution.
Side effects: may process data when executed (read-only or write to target).
Dependencies: source access, target access.
Stop condition: architecture saved; quality rules defined.
Risk: medium — data corruption risks; requires testing.
Boundary: defines pipeline; execution requires approval.

ETL Pipeline Design

Design an ETL / ELT pipeline — extraction, transformation, load — with reproducible steps, schema evolution, and data quality checks.

Process

1. Source analysis

Source type: database (SQL / NoSQL), file (CSV / JSON / Parquet / ORC), API (REST / GraphQL / gRPC), stream (Kafka / Kinesis / Pub/Sub).
Schema: columns, types, constraints.
Frequency: batch (hourly / daily) or streaming (near real-time).
Volume: rows / GB per run.
Quality issues: missing values, out-of-range, duplicates, format errors.

Completion criterion: source profile saved.

2. Transformation design

Filter: remove rows that don't meet criteria.
Clean: fix types, trim whitespace, standardise formats.
Aggregate: group by key, compute statistics.
Enrich: join with reference data (customer, product, geography).
Deduplicate: exact / fuzzy / probabilistic.

Completion criterion: transformation steps documented.

3. Load design

Target type: warehouse (Snowflake / BigQuery / Redshift / Databricks) vs database vs lake.
Load strategy: full refresh (replace all) / incremental (append new / update changed) / merge (upsert).
Partition / clustering strategy for performance.
Backfill plan for historical data.

Completion criterion: load strategy saved.

4. Data quality

Null rate per column (threshold: < X%).
Out-of-range checks (min / max / expected range).
Duplicate rate.
Referential integrity (foreign key consistency).
Freshness (data delay from source to target).

Completion criterion: quality rules saved with thresholds and actions (fail / warn / fix).

5. Schema evolution

How to handle new columns (add, ignore, raise error).
How to handle renamed / deleted columns.
Versioning: schema registry (e.g. Confluent Schema Registry, AWS Glue Data Catalog).

Completion criterion: evolution plan saved.

---name: data-etl-pipelinedescription: Design ETL / ELT pipelines — extraction, transformation, load — with reproducible steps, schema evolution, quality checks, and observability.---## Contract- **Input:** source data descriptions, target schema, quality requirements.- **Output:** pipeline architecture + quality rules + schema evolution.- **Side effects:** may process data when executed (read-only or write to target).- **Dependencies:** source access, target access.- **Stop condition:** architecture saved; quality rules defined.- **Risk:** medium — data corruption risks; requires testing.- **Boundary:** defines pipeline; execution requires approval.# ETL Pipeline DesignDesign an **ETL / ELT pipeline** — extraction, transformation, load — with reproducible steps, schema evolution, and data quality checks.## Process### 1. Source analysis- Source type: database (SQL / NoSQL), file (CSV / JSON / Parquet / ORC), API (REST / GraphQL / gRPC), stream (Kafka / Kinesis / Pub/Sub).- Schema: columns, types, constraints.- Frequency: batch (hourly / daily) or streaming (near real-time).- Volume: rows / GB per run.- Quality issues: missing values, out-of-range, duplicates, format errors.**Completion criterion:** source profile saved.### 2. Transformation design- Filter: remove rows that don't meet criteria.- Clean: fix types, trim whitespace, standardise formats.- Aggregate: group by key, compute statistics.- Enrich: join with reference data (customer, product, geography).- Deduplicate: exact / fuzzy / probabilistic.**Completion criterion:** transformation steps documented.### 3. Load design- Target type: warehouse (Snowflake / BigQuery / Redshift / Databricks) vs database vs lake.- Load strategy: full refresh (replace all) / incremental (append new / update changed) / merge (upsert).- Partition / clustering strategy for performance.- Backfill plan for historical data.**Completion criterion:** load strategy saved.### 4. Data quality- Null rate per column (threshold: < X%).- Out-of-range checks (min / max / expected range).- Duplicate rate.- Referential integrity (foreign key consistency).- Freshness (data delay from source to target).**Completion criterion:** quality rules saved with thresholds and actions (fail / warn / fix).### 5. Schema evolution- How to handle new columns (add, ignore, raise error).- How to handle renamed / deleted columns.- Versioning: schema registry (e.g. Confluent Schema Registry, AWS Glue Data Catalog).**Completion criterion:** evolution plan saved.

Frequently asked questions

Data Etl Pipeline