Data Engineering
When to use
- Design data pipelines and ETL/ELT
- Optimize Apache Spark jobs
- Build dbt transformation models
- Create Airflow DAGs
- Implement data quality frameworks
- Set up streaming architectures
Tools
- Apache Spark: partitioning, caching, shuffle optimization
- dbt: model organization, testing, incremental strategies
- Airflow: DAG patterns, operators, sensors
- Data quality: Great Expectations, dbt tests, data contracts
Architecture
- Batch vs streaming vs hybrid
- Data lakehouse patterns
- Medallion architecture (bronze/silver/gold)
- Change data capture (CDC)
- Schema evolution strategies