Senior Data Engineer
Production-grade data engineering skill for building scalable, reliable data systems.
Table of Contents
- Trigger Phrases
- Quick Start
- Workflows
- Architecture Decision Framework
- Tech Stack
- Reference Documentation
- Troubleshooting
Trigger Phrases
Activate this skill when you see:
Pipeline Design:
- "Design a data pipeline for..."
- "Build an ETL/ELT process..."
- "How should I ingest data from..."
- "Set up data extraction from..."
Architecture:
- "Should I use batch or streaming?"
- "Lambda vs Kappa architecture"
- "How to handle late-arriving data"
- "Design a data lakehouse"
Data Modeling:
- "Create a dimensional model..."
- "Star schema vs snowflake"
- "Implement slowly changing dimensions"
- "Design a data vault"
Data Quality:
- "Add data validation to..."
- "Set up data quality checks"
- "Monitor data freshness"
- "Implement data contracts"
Performance:
- "Optimize this Spark job"
- "Query is running slow"
- "Reduce pipeline execution time"
- "Tune Airflow DAG"
Quick Start
Core Tools
# Generate pipeline orchestration config
python scripts/pipeline_orchestrator.py generate \
--type airflow \
--source postgres \
--destination snowflake \
--schedule "0 5 * * *"
# Validate data quality
python scripts/data_quality_validator.py validate \
--input data/sales.parquet \
--schema schemas/sales.json \
--checks freshness,completeness,uniqueness
# Optimize ETL performance
python scripts/etl_performance_optimizer.py analyze \
--query queries/daily_aggregation.sql \
--engine spark \
--recommend
Workflows
→ See references/workflows.md for details
Architecture Decision Framework
Use this framework to choose the right approach for your data pipeline.
Batch vs Streaming
| Criteria |
Batch |
Streaming |
| Latency requirement |
Hours to days |
Seconds to minutes |
| Data volume |
Large historical datasets |
Continuous event streams |
| Processing complexity |
Complex transformations, ML |
Simple aggregations, filtering |
| Cost sensitivity |
More cost-effective |
Higher infrastructure cost |
| Error handling |
Easier to reprocess |
Requires careful design |
Decision Tree:
Is real-time insight required?
├── Yes → Use streaming
│ └── Is exactly-once semantics needed?
│ ├── Yes → Kafka + Flink/Spark Structured Streaming
│ └── No → Kafka + consumer groups
└── No → Use batch
└── Is data volume > 1TB daily?
├── Yes → Spark/Databricks
└── No → dbt + warehouse compute
Lambda vs Kappa Architecture
| Aspect |
Lambda |
Kappa |
| Complexity |
Two codebases (batch + stream) |
Single codebase |
| Maintenance |
Higher (sync batch/stream logic) |
Lower |
| Reprocessing |
Native batch layer |
Replay from source |
| Use case |
ML training + real-time serving |
Pure event-driven |
When to choose Lambda:
- Need to train ML models on historical data
- Complex batch transformations not feasible in streaming
- Existing batch infrastructure
When to choose Kappa:
- Event-sourced architecture
- All processing can be expressed as stream operations
- Starting fresh without legacy systems
Data Warehouse vs Data Lakehouse
| Feature |
Warehouse (Snowflake/BigQuery) |
Lakehouse (Delta/Iceberg) |
| Best for |
BI, SQL analytics |
ML, unstructured data |
| Storage cost |
Higher (proprietary format) |
Lower (open formats) |
| Flexibility |
Schema-on-write |
Schema-on-read |
| Performance |
Excellent for SQL |
Good, improving |
| Ecosystem |
Mature BI tools |
Growing ML tooling |
Tech Stack
| Category |
Technologies |
| Languages |
Python, SQL, Scala |
| Orchestration |
Airflow, Prefect, Dagster |
| Transformation |
dbt, Spark, Flink |
| Streaming |
Kafka, Kinesis, Pub/Sub |
| Storage |
S3, GCS, Delta Lake, Iceberg |
| Warehouses |
Snowflake, BigQuery, Redshift, Databricks |
| Quality |
Great Expectations, dbt tests, Monte Carlo |
| Monitoring |
Prometheus, Grafana, Datadog |
Reference Documentation
1. Data Pipeline Architecture
See references/data_pipeline_architecture.md for:
- Lambda vs Kappa architecture patterns
- Batch processing with Spark and Airflow
- Stream processing with Kafka and Flink
- Exactly-once semantics implementation
- Error handling and dead letter queues
2. Data Modeling Patterns
See references/data_modeling_patterns.md for:
- Dimensional modeling (Star/Snowflake)
- Slowly Changing Dimensions (SCD Types 1-6)
- Data Vault modeling
- dbt best practices
- Partitioning and clustering
3. DataOps Best Practices
See references/dataops_best_practices.md for:
- Data testing frameworks
- Data contracts and schema validation
- CI/CD for data pipelines
- Observability and lineage
- Incident response
Troubleshooting
→ See references/troubleshooting.md for details
1---2name: "senior-data-engineer"3description: "Data pipelines: Airflow DAGs, Spark, dbt, Kafka, ETL/ELT, data quality checks. Load whenever a DAG or batch job times out or fails, data arrives wrong, or the user designs an ingestion pipeline."4metadata:5 source: https://skillmd.com/skills/alirezarezvani/senior-data-engineer6---78# Senior Data Engineer910Production-grade data engineering skill for building scalable, reliable data systems.1112## Table of Contents13141. [Trigger Phrases](#trigger-phrases)152. [Quick Start](#quick-start)163. [Workflows](#workflows)174. [Architecture Decision Framework](#architecture-decision-framework)185. [Tech Stack](#tech-stack)196. [Reference Documentation](#reference-documentation)207. [Troubleshooting](#troubleshooting)2122---2324## Trigger Phrases2526Activate this skill when you see:2728**Pipeline Design:**29- "Design a data pipeline for..."30- "Build an ETL/ELT process..."31- "How should I ingest data from..."32- "Set up data extraction from..."3334**Architecture:**35- "Should I use batch or streaming?"36- "Lambda vs Kappa architecture"37- "How to handle late-arriving data"38- "Design a data lakehouse"3940**Data Modeling:**41- "Create a dimensional model..."42- "Star schema vs snowflake"43- "Implement slowly changing dimensions"44- "Design a data vault"4546**Data Quality:**47- "Add data validation to..."48- "Set up data quality checks"49- "Monitor data freshness"50- "Implement data contracts"5152**Performance:**53- "Optimize this Spark job"54- "Query is running slow"55- "Reduce pipeline execution time"56- "Tune Airflow DAG"5758---5960## Quick Start6162### Core Tools6364```bash65# Generate pipeline orchestration config66python scripts/pipeline_orchestrator.py generate \67 --type airflow \68 --source postgres \69 --destination snowflake \70 --schedule "0 5 * * *"7172# Validate data quality73python scripts/data_quality_validator.py validate \74 --input data/sales.parquet \75 --schema schemas/sales.json \76 --checks freshness,completeness,uniqueness7778# Optimize ETL performance79python scripts/etl_performance_optimizer.py analyze \80 --query queries/daily_aggregation.sql \81 --engine spark \82 --recommend83```8485---8687## Workflows88→ See references/workflows.md for details8990## Architecture Decision Framework9192Use this framework to choose the right approach for your data pipeline.9394### Batch vs Streaming9596| Criteria | Batch | Streaming |97|----------|-------|-----------|98| **Latency requirement** | Hours to days | Seconds to minutes |99| **Data volume** | Large historical datasets | Continuous event streams |100| **Processing complexity** | Complex transformations, ML | Simple aggregations, filtering |101| **Cost sensitivity** | More cost-effective | Higher infrastructure cost |102| **Error handling** | Easier to reprocess | Requires careful design |103104**Decision Tree:**105```106Is real-time insight required?107├── Yes → Use streaming108│ └── Is exactly-once semantics needed?109│ ├── Yes → Kafka + Flink/Spark Structured Streaming110│ └── No → Kafka + consumer groups111└── No → Use batch112 └── Is data volume > 1TB daily?113 ├── Yes → Spark/Databricks114 └── No → dbt + warehouse compute115```116117### Lambda vs Kappa Architecture118119| Aspect | Lambda | Kappa |120|--------|--------|-------|121| **Complexity** | Two codebases (batch + stream) | Single codebase |122| **Maintenance** | Higher (sync batch/stream logic) | Lower |123| **Reprocessing** | Native batch layer | Replay from source |124| **Use case** | ML training + real-time serving | Pure event-driven |125126**When to choose Lambda:**127- Need to train ML models on historical data128- Complex batch transformations not feasible in streaming129- Existing batch infrastructure130131**When to choose Kappa:**132- Event-sourced architecture133- All processing can be expressed as stream operations134- Starting fresh without legacy systems135136### Data Warehouse vs Data Lakehouse137138| Feature | Warehouse (Snowflake/BigQuery) | Lakehouse (Delta/Iceberg) |139|---------|-------------------------------|---------------------------|140| **Best for** | BI, SQL analytics | ML, unstructured data |141| **Storage cost** | Higher (proprietary format) | Lower (open formats) |142| **Flexibility** | Schema-on-write | Schema-on-read |143| **Performance** | Excellent for SQL | Good, improving |144| **Ecosystem** | Mature BI tools | Growing ML tooling |145146---147148## Tech Stack149150| Category | Technologies |151|----------|--------------|152| **Languages** | Python, SQL, Scala |153| **Orchestration** | Airflow, Prefect, Dagster |154| **Transformation** | dbt, Spark, Flink |155| **Streaming** | Kafka, Kinesis, Pub/Sub |156| **Storage** | S3, GCS, Delta Lake, Iceberg |157| **Warehouses** | Snowflake, BigQuery, Redshift, Databricks |158| **Quality** | Great Expectations, dbt tests, Monte Carlo |159| **Monitoring** | Prometheus, Grafana, Datadog |160161---162163## Reference Documentation164165### 1. Data Pipeline Architecture166See `references/data_pipeline_architecture.md` for:167- Lambda vs Kappa architecture patterns168- Batch processing with Spark and Airflow169- Stream processing with Kafka and Flink170- Exactly-once semantics implementation171- Error handling and dead letter queues172173### 2. Data Modeling Patterns174See `references/data_modeling_patterns.md` for:175- Dimensional modeling (Star/Snowflake)176- Slowly Changing Dimensions (SCD Types 1-6)177- Data Vault modeling178- dbt best practices179- Partitioning and clustering180181### 3. DataOps Best Practices182See `references/dataops_best_practices.md` for:183- Data testing frameworks184- Data contracts and schema validation185- CI/CD for data pipelines186- Observability and lineage187- Incident response188189---190191## Troubleshooting192→ See references/troubleshooting.md for details193