# Data Engineering

> Use for data pipeline design, Apache Spark optimization, dbt transformations, Airflow DAGs, data quality frameworks (Great Expectations), streaming architectures, and modern data stack implementation.

- Skill: `henryhawke/data-engineering` (Agent Skill, multi-file: 6 files)
- Install (CLI): `npx skillmds@latest add henryhawke/data-engineering`
- Raw SKILL.md: https://api.skillmd.com/api/skills/henryhawke/data-engineering/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: henryhawke (https://skillmd.com/u/henryhawke)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/henryhawke/data-engineering

---


# Data Engineering

## When to use
- Design data pipelines and ETL/ELT
- Optimize Apache Spark jobs
- Build dbt transformation models
- Create Airflow DAGs
- Implement data quality frameworks
- Set up streaming architectures

## Tools
- Apache Spark: partitioning, caching, shuffle optimization
- dbt: model organization, testing, incremental strategies
- Airflow: DAG patterns, operators, sensors
- Data quality: Great Expectations, dbt tests, data contracts

## Architecture
- Batch vs streaming vs hybrid
- Data lakehouse patterns
- Medallion architecture (bronze/silver/gold)
- Change data capture (CDC)
- Schema evolution strategies