How do I install the Data Engineering skill?

Run npx skillmds@latest add irahardianto/data-engineering in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Data Engineering safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Data Engineering?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Data Engineering free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Data Engineering?

irahardianto (@irahardianto) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Data Engineering by irahardianto — Agent Skill

Name: Data Engineering
Author: irahardianto

File contents

Data Engineering Principles

Guidelines for building reliable, scalable data pipelines and platforms.

When to Invoke

Designing data pipelines (ETL/ELT)
Evaluating batch vs stream processing
Data quality and governance requirements
Data warehouse/lake architecture decisions

Pipeline Architecture

Design Principles

Idempotent pipelines — re-running produces same result. Use upserts, not inserts.
Schema evolution — handle new fields without breaking consumers.
Exactly-once processing — deduplication at ingestion, idempotency keys.
Incremental processing — process only new/changed data, not full reloads.

Patterns

Pattern	When to Use
Batch ETL	Scheduled, high volume, latency-tolerant
Streaming	Real-time, event-driven, low latency
Lambda	Both batch and stream (complexity trade-off)
Kappa	Stream-only, reprocessing via replay
Medallion	Bronze (raw) → Silver (cleaned) → Gold (curated)

Data Quality

Checks (Non-Negotiable)

Completeness — no unexpected nulls in required fields
Uniqueness — no duplicate records on primary keys
Referential integrity — foreign keys resolve
Freshness — data arrives within SLA window
Volume — row counts within expected range (±threshold)

Framework

Source → Validate (schema, nulls, types) → Transform → Validate (business rules) → Load → Verify (counts, checksums)

Orchestration

Tool	Strength
Apache Airflow	Most mature, Python-native, DAG-based
Dagster	Type-safe, asset-oriented, modern
Prefect	Pythonic, flow-based, cloud-native

Best Practices

DAGs should be idempotent and retriable
Separate orchestration from computation
Use backfill capabilities for historical reprocessing
Alert on SLA breaches, not just failures

Data Modeling

Model	When
Star schema	Analytics, BI dashboards, simple queries
Data Vault	Enterprise, auditability, multiple sources
Dimensional	Aggregated reporting, OLAP

Governance

Data lineage tracked (source → transformation → destination)
Access controls per dataset/table
PII identified and masked/encrypted
Retention policies documented and automated

Database Design Principles @.gemini/skills/database-design-principles/SKILL.md
SQL Idioms @.gemini/skills/sql-idioms/SKILL.md
Logging and Observability Principles @.gemini/skills/logging-and-observability-principles/SKILL.md

---name: data-engineeringdescription: Data pipeline architecture, ETL/ELT patterns, data quality, batch vs stream processing, orchestration, and data governance principles.---# Data Engineering PrinciplesGuidelines for building reliable, scalable data pipelines and platforms.## When to Invoke- Designing data pipelines (ETL/ELT)- Evaluating batch vs stream processing- Data quality and governance requirements- Data warehouse/lake architecture decisions## Pipeline Architecture### Design Principles1. **Idempotent pipelines** — re-running produces same result. Use upserts, not inserts.2. **Schema evolution** — handle new fields without breaking consumers.3. **Exactly-once processing** — deduplication at ingestion, idempotency keys.4. **Incremental processing** — process only new/changed data, not full reloads.### Patterns| Pattern | When to Use ||---|---|| **Batch ETL** | Scheduled, high volume, latency-tolerant || **Streaming** | Real-time, event-driven, low latency || **Lambda** | Both batch and stream (complexity trade-off) || **Kappa** | Stream-only, reprocessing via replay || **Medallion** | Bronze (raw) → Silver (cleaned) → Gold (curated) |## Data Quality### Checks (Non-Negotiable)- **Completeness** — no unexpected nulls in required fields- **Uniqueness** — no duplicate records on primary keys- **Referential integrity** — foreign keys resolve- **Freshness** — data arrives within SLA window- **Volume** — row counts within expected range (±threshold)### Framework```Source → Validate (schema, nulls, types) → Transform → Validate (business rules) → Load → Verify (counts, checksums)```## Orchestration| Tool | Strength ||---|---|| Apache Airflow | Most mature, Python-native, DAG-based || Dagster | Type-safe, asset-oriented, modern || Prefect | Pythonic, flow-based, cloud-native |### Best Practices- DAGs should be idempotent and retriable- Separate orchestration from computation- Use backfill capabilities for historical reprocessing- Alert on SLA breaches, not just failures## Data Modeling| Model | When ||---|---|| **Star schema** | Analytics, BI dashboards, simple queries || **Data Vault** | Enterprise, auditability, multiple sources || **Dimensional** | Aggregated reporting, OLAP |## Governance- Data lineage tracked (source → transformation → destination)- Access controls per dataset/table- PII identified and masked/encrypted- Retention policies documented and automated## Related- Database Design Principles @.gemini/skills/database-design-principles/SKILL.md- SQL Idioms @.gemini/skills/sql-idioms/SKILL.md- Logging and Observability Principles @.gemini/skills/logging-and-observability-principles/SKILL.md

Frequently asked questions

Data Engineering