Senior Data Engineer
Production-grade data engineering skill for building scalable, reliable data systems.
Table of Contents
- Trigger Phrases
- Quick Start
- Workflows
- Architecture Decision Framework
- Tech Stack
- Reference Documentation
- Troubleshooting
Trigger Phrases
Activate this skill when you see:
Pipeline Design:
- "Design a data pipeline for..."
- "Build an ETL/ELT process..."
- "How should I ingest data from..."
- "Set up data extraction from..."
Architecture:
- "Should I use batch or streaming?"
- "Lambda vs Kappa architecture"
- "How to handle late-arriving data"
- "Design a data lakehouse"
Data Modeling:
- "Create a dimensional model..."
- "Star schema vs snowflake"
- "Implement slowly changing dimensions"
- "Design a data vault"
Data Quality:
- "Add data validation to..."
- "Set up data quality checks"
- "Monitor data freshness"
- "Implement data contracts"
Performance:
- "Optimize this Spark job"
- "Query is running slow"
- "Reduce pipeline execution time"
- "Tune Airflow DAG"
Quick Start
Core Tools
# Generate pipeline orchestration config
python scripts/pipeline_orchestrator.py generate \
--type airflow \
--source postgres \
--destination snowflake \
--schedule "0 5 * * *"
# Validate data quality
python scripts/data_quality_validator.py validate \
--input data/sales.parquet \
--schema schemas/sales.json \
--checks freshness,completeness,uniqueness
# Optimize ETL performance
python scripts/etl_performance_optimizer.py analyze \
--query queries/daily_aggregation.sql \
--engine spark \
--recommend
Workflows
→ See references/workflows.md for details
Architecture Decision Framework
Use this framework to choose the right approach for your data pipeline.
Batch vs Streaming
| Criteria |
Batch |
Streaming |
| Latency requirement |
Hours to days |
Seconds to minutes |
| Data volume |
Large historical datasets |
Continuous event streams |
| Processing complexity |
Complex transformations, ML |
Simple aggregations, filtering |
| Cost sensitivity |
More cost-effective |
Higher infrastructure cost |
| Error handling |
Easier to reprocess |
Requires careful design |
Decision Tree:
Is real-time insight required?
├── Yes → Use streaming
│ └── Is exactly-once semantics needed?
│ ├── Yes → Kafka + Flink/Spark Structured Streaming
│ └── No → Kafka + consumer groups
└── No → Use batch
└── Is data volume > 1TB daily?
├── Yes → Spark/Databricks
└── No → dbt + warehouse compute
Lambda vs Kappa Architecture
| Aspect |
Lambda |
Kappa |
| Complexity |
Two codebases (batch + stream) |
Single codebase |
| Maintenance |
Higher (sync batch/stream logic) |
Lower |
| Reprocessing |
Native batch layer |
Replay from source |
| Use case |
ML training + real-time serving |
Pure event-driven |
When to choose Lambda:
- Need to train ML models on historical data
- Complex batch transformations not feasible in streaming
- Existing batch infrastructure
When to choose Kappa:
- Event-sourced architecture
- All processing can be expressed as stream operations
- Starting fresh without legacy systems
Data Warehouse vs Data Lakehouse
| Feature |
Warehouse (Snowflake/BigQuery) |
Lakehouse (Delta/Iceberg) |
| Best for |
BI, SQL analytics |
ML, unstructured data |
| Storage cost |
Higher (proprietary format) |
Lower (open formats) |
| Flexibility |
Schema-on-write |
Schema-on-read |
| Performance |
Excellent for SQL |
Good, improving |
| Ecosystem |
Mature BI tools |
Growing ML tooling |
Tech Stack
| Category |
Technologies |
| Languages |
Python, SQL, Scala |
| Orchestration |
Airflow, Prefect, Dagster |
| Transformation |
dbt, Spark, Flink |
| Streaming |
Kafka, Kinesis, Pub/Sub |
| Storage |
S3, GCS, Delta Lake, Iceberg |
| Warehouses |
Snowflake, BigQuery, Redshift, Databricks |
| Quality |
Great Expectations, dbt tests, Monte Carlo |
| Monitoring |
Prometheus, Grafana, Datadog |
Reference Documentation
1. Data Pipeline Architecture
See references/data_pipeline_architecture.md for:
- Lambda vs Kappa architecture patterns
- Batch processing with Spark and Airflow
- Stream processing with Kafka and Flink
- Exactly-once semantics implementation
- Error handling and dead letter queues
2. Data Modeling Patterns
See references/data_modeling_patterns.md for:
- Dimensional modeling (Star/Snowflake)
- Slowly Changing Dimensions (SCD Types 1-6)
- Data Vault modeling
- dbt best practices
- Partitioning and clustering
3. DataOps Best Practices
See references/dataops_best_practices.md for:
- Data testing frameworks
- Data contracts and schema validation
- CI/CD for data pipelines
- Observability and lineage
- Incident response
Troubleshooting
→ See references/troubleshooting.md for details
1---2name: senior-data-engineer3description: Data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps. Use when designing data architectures, building data pipelines, optimizing data workflows, implementing data governance, or troubleshooting data issues.4metadata:5 source: https://skillmd.com/skills/bestagentkits/senior-data-engineer6---78# Senior Data Engineer910Production-grade data engineering skill for building scalable, reliable data systems.1112## Table of Contents13141. [Trigger Phrases](#trigger-phrases)152. [Quick Start](#quick-start)163. [Workflows](#workflows)174. [Architecture Decision Framework](#architecture-decision-framework)185. [Tech Stack](#tech-stack)196. [Reference Documentation](#reference-documentation)207. [Troubleshooting](#troubleshooting)2122---2324## Trigger Phrases2526Activate this skill when you see:2728**Pipeline Design:**29- "Design a data pipeline for..."30- "Build an ETL/ELT process..."31- "How should I ingest data from..."32- "Set up data extraction from..."3334**Architecture:**35- "Should I use batch or streaming?"36- "Lambda vs Kappa architecture"37- "How to handle late-arriving data"38- "Design a data lakehouse"3940**Data Modeling:**41- "Create a dimensional model..."42- "Star schema vs snowflake"43- "Implement slowly changing dimensions"44- "Design a data vault"4546**Data Quality:**47- "Add data validation to..."48- "Set up data quality checks"49- "Monitor data freshness"50- "Implement data contracts"5152**Performance:**53- "Optimize this Spark job"54- "Query is running slow"55- "Reduce pipeline execution time"56- "Tune Airflow DAG"5758---5960## Quick Start6162### Core Tools6364```bash65# Generate pipeline orchestration config66python scripts/pipeline_orchestrator.py generate \67 --type airflow \68 --source postgres \69 --destination snowflake \70 --schedule "0 5 * * *"7172# Validate data quality73python scripts/data_quality_validator.py validate \74 --input data/sales.parquet \75 --schema schemas/sales.json \76 --checks freshness,completeness,uniqueness7778# Optimize ETL performance79python scripts/etl_performance_optimizer.py analyze \80 --query queries/daily_aggregation.sql \81 --engine spark \82 --recommend83```8485---8687## Workflows88→ See references/workflows.md for details8990## Architecture Decision Framework9192Use this framework to choose the right approach for your data pipeline.9394### Batch vs Streaming9596| Criteria | Batch | Streaming |97|----------|-------|-----------|98| **Latency requirement** | Hours to days | Seconds to minutes |99| **Data volume** | Large historical datasets | Continuous event streams |100| **Processing complexity** | Complex transformations, ML | Simple aggregations, filtering |101| **Cost sensitivity** | More cost-effective | Higher infrastructure cost |102| **Error handling** | Easier to reprocess | Requires careful design |103104**Decision Tree:**105```106Is real-time insight required?107├── Yes → Use streaming108│ └── Is exactly-once semantics needed?109│ ├── Yes → Kafka + Flink/Spark Structured Streaming110│ └── No → Kafka + consumer groups111└── No → Use batch112 └── Is data volume > 1TB daily?113 ├── Yes → Spark/Databricks114 └── No → dbt + warehouse compute115```116117### Lambda vs Kappa Architecture118119| Aspect | Lambda | Kappa |120|--------|--------|-------|121| **Complexity** | Two codebases (batch + stream) | Single codebase |122| **Maintenance** | Higher (sync batch/stream logic) | Lower |123| **Reprocessing** | Native batch layer | Replay from source |124| **Use case** | ML training + real-time serving | Pure event-driven |125126**When to choose Lambda:**127- Need to train ML models on historical data128- Complex batch transformations not feasible in streaming129- Existing batch infrastructure130131**When to choose Kappa:**132- Event-sourced architecture133- All processing can be expressed as stream operations134- Starting fresh without legacy systems135136### Data Warehouse vs Data Lakehouse137138| Feature | Warehouse (Snowflake/BigQuery) | Lakehouse (Delta/Iceberg) |139|---------|-------------------------------|---------------------------|140| **Best for** | BI, SQL analytics | ML, unstructured data |141| **Storage cost** | Higher (proprietary format) | Lower (open formats) |142| **Flexibility** | Schema-on-write | Schema-on-read |143| **Performance** | Excellent for SQL | Good, improving |144| **Ecosystem** | Mature BI tools | Growing ML tooling |145146---147148## Tech Stack149150| Category | Technologies |151|----------|--------------|152| **Languages** | Python, SQL, Scala |153| **Orchestration** | Airflow, Prefect, Dagster |154| **Transformation** | dbt, Spark, Flink |155| **Streaming** | Kafka, Kinesis, Pub/Sub |156| **Storage** | S3, GCS, Delta Lake, Iceberg |157| **Warehouses** | Snowflake, BigQuery, Redshift, Databricks |158| **Quality** | Great Expectations, dbt tests, Monte Carlo |159| **Monitoring** | Prometheus, Grafana, Datadog |160161---162163## Reference Documentation164165### 1. Data Pipeline Architecture166See `references/data_pipeline_architecture.md` for:167- Lambda vs Kappa architecture patterns168- Batch processing with Spark and Airflow169- Stream processing with Kafka and Flink170- Exactly-once semantics implementation171- Error handling and dead letter queues172173### 2. Data Modeling Patterns174See `references/data_modeling_patterns.md` for:175- Dimensional modeling (Star/Snowflake)176- Slowly Changing Dimensions (SCD Types 1-6)177- Data Vault modeling178- dbt best practices179- Partitioning and clustering180181### 3. DataOps Best Practices182See `references/dataops_best_practices.md` for:183- Data testing frameworks184- Data contracts and schema validation185- CI/CD for data pipelines186- Observability and lineage187- Incident response188189---190191## Troubleshooting192→ See references/troubleshooting.md for details193