# Senior Data Engineer

> World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps. Use when designing data architectures, building data pipelines, optimizing data workflows, or implementing data governance.

- Skill: `gurdasnijor/senior-data-engineer` (Agent Skill, multi-file: 11 files)
- Install (CLI): `npx skillmds@latest add gurdasnijor/senior-data-engineer`
- Raw SKILL.md: https://api.skillmd.com/api/skills/gurdasnijor/senior-data-engineer/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- License: MIT
- Author: gurdasnijor (https://skillmd.com/u/gurdasnijor)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/gurdasnijor/senior-data-engineer

---


# Senior Data Engineer

## Core Capabilities

- **[Capability 1]** - [Description]
- **[Capability 2]** - [Description]
- **[Capability 3]** - [Description]
- **[Capability 4]** - [Description]


## Key Workflows

### Workflow 1: [Workflow Name]

**Time:** [Duration estimate]

**Steps:**
1. [Step 1]
2. [Step 2]
3. [Step 3]

**Expected Output:** [What success looks like]

### Workflow 2: [Workflow Name]

**Time:** [Duration estimate]

**Steps:**
1. [Step 1]
2. [Step 2]
3. [Step 3]

**Expected Output:** [What success looks like]


World-class data engineering for production-grade data systems, scalable pipelines, and enterprise data platforms.

## Overview

This skill provides comprehensive expertise in data engineering fundamentals through advanced production patterns. From designing medallion architectures to implementing real-time streaming pipelines, it covers the full spectrum of modern data engineering including ETL/ELT design, data quality frameworks, pipeline orchestration, and DataOps practices.

**What This Skill Provides:**
- Production-ready pipeline templates (Airflow, Spark, dbt)
- Comprehensive data quality validation framework
- Performance optimization and cost analysis tools
- Data architecture patterns (Lambda, Kappa, Medallion)
- Complete DataOps CI/CD workflows

**Best For:**
- Building scalable data pipelines for enterprise systems
- Implementing data quality and governance frameworks
- Optimizing ETL performance and cloud costs
- Designing modern data architectures (lake, warehouse, lakehouse)
- Production ML/AI data infrastructure

## Quick Start

### Pipeline Orchestration

```bash
# Generate Airflow DAG from configuration
python scripts/pipeline_orchestrator.py --config pipeline_config.yaml --output dags/

# Validate pipeline configuration
python scripts/pipeline_orchestrator.py --config pipeline_config.yaml --validate

# Use incremental load template
python scripts/pipeline_orchestrator.py --template incremental --output dags/
```

### Data Quality Validation

```bash
# Validate CSV file with quality checks
python scripts/data_quality_validator.py --input data/sales.csv --output report.html

# Validate database table with custom rules
python scripts/data_quality_validator.py \
    --connection postgresql://user:pass@host/db \
    --table sales_transactions \
    --rules rules/sales_validation.yaml \
    --threshold 0.95
```

### Performance Optimization

```bash
# Analyze pipeline performance and get recommendations
python scripts/etl_performance_optimizer.py \
    --airflow-db postgresql://host/airflow \
    --dag-id sales_etl_pipeline \
    --days 30 \
    --optimize

# Analyze Spark job performance
python scripts/etl_performance_optimizer.py \
    --spark-history-server http://spark-history:18080 \
    --app-id app-20250115-001
```

## Core Workflows

### 1. Building Production Data Pipelines

**Steps:**
1. **Design Architecture:** Choose pattern (Lambda, Kappa, Medallion) based on requirements
2. **Configure Pipeline:** Create YAML configuration with sources, transformations, targets
3. **Generate DAG:** `python scripts/pipeline_orchestrator.py --config config.yaml`
4. **Add Quality Checks:** Define validation rules for data quality
5. **Deploy & Monitor:** Deploy to Airflow, configure alerts, track metrics

**Pipeline Patterns:** See [frameworks.md](references/frameworks.md) for Lambda Architecture, Kappa Architecture, Medallion Architecture (Bronze/Silver/Gold), and Microservices Data patterns.

**Templates:** See [templates.md](references/templates.md) for complete Airflow DAG templates, Spark job templates, dbt models, and Docker configurations.

### 2. Data Quality Management

**Steps:**
1. **Define Rules:** Create validation rules covering completeness, accuracy, consistency
2. **Run Validation:** `python scripts/data_quality_validator.py --rules rules.yaml`
3. **Review Results:** Analyze quality scores and failed checks
4. **Integrate CI/CD:** Add validation to pipeline deployment process
5. **Monitor Trends:** Track quality scores over time

**Quality Framework:** See [frameworks.md](references/frameworks.md) for complete Data Quality Framework covering all dimensions (completeness, accuracy, consistency, timeliness, validity).

**Validation Templates:** See [templates.md](references/templates.md) for validation configuration examples and Python API usage.

### 3. Data Modeling & Transformation

**Steps:**
1. **Choose Modeling Approach:** Dimensional (Kimball), Data Vault 2.0, or One Big Table
2. **Design Schema:** Define fact tables, dimensions, and relationships
3. **Implement with dbt:** Create staging, intermediate, and mart models
4. **Handle SCD:** Implement slowly changing dimension logic (Type 1/2/3)
5. **Test & Deploy:** Run dbt tests, generate documentation, deploy

**Modeling Patterns:** See [frameworks.md](references/frameworks.md) for Dimensional Modeling (Kimball), Data Vault 2.0, One Big Table (OBT), and SCD implementations.

**dbt Templates:** See [templates.md](references/templates.md) for complete dbt model templates including staging, intermediate, fact tables, and SCD Type 2 logic.

### 4. Performance Optimization

**Steps:**
1. **Profile Pipeline:** Run performance analyzer on recent pipeline executions
2. **Identify Bottlenecks:** Review execution time breakdown and slow tasks
3. **Apply Optimizations:** Implement recommendations (partitioning, indexing, batching)
4. **Tune Spark Jobs:** Optimize memory, parallelism, and shuffle settings
5. **Measure Impact:** Compare before/after metrics, track cost savings

**Optimization Strategies:** See [frameworks.md](references/frameworks.md) for performance best practices including partitioning strategies, query optimization, and Spark tuning.

**Analysis Tools:** See [tools.md](references/tools.md) for complete documentation on etl_performance_optimizer.py with query analysis and Spark tuning.

## Python Tools

### pipeline_orchestrator.py

Automated Airflow DAG generation with intelligent dependency resolution and monitoring.

**Key Features:**
- Generate production-ready DAGs from YAML configuration
- Automatic task dependency resolution
- Built-in retry logic and error handling
- Multi-source support (PostgreSQL, S3, BigQuery, Snowflake)
- Integrated quality checks and alerting

**Usage:**
```bash
# Basic DAG generation
python scripts/pipeline_orchestrator.py --config pipeline_config.yaml --output dags/

# With validation
python scripts/pipeline_orchestrator.py --config config.yaml --validate

# From template
python scripts/pipeline_orchestrator.py --template incremental --output dags/
```

**Complete Documentation:** See [tools.md](references/tools.md) for full configuration options, templates, and integration examples.

### data_quality_validator.py

Comprehensive data quality validation framework with automated checks and reporting.

**Capabilities:**
- Multi-dimensional validation (completeness, accuracy, consistency, timeliness, validity)
- Great Expectations integration
- Custom business rule validation
- HTML/PDF report generation
- Anomaly detection
- Historical trend tracking

**Usage:**
```bash
# Validate with custom rules
python scripts/data_quality_validator.py \
    --input data/sales.csv \
    --rules rules/sales_validation.yaml \
    --output report.html

# Database table validation
python scripts/data_quality_validator.py \
    --connection postgresql://host/db \
    --table sales_transactions \
    --threshold 0.95
```

**Complete Documentation:** See [tools.md](references/tools.md) for rule configuration, API usage, and integration patterns.

### etl_performance_optimizer.py

Pipeline performance analysis with actionable optimization recommendations.

**Capabilities:**
- Airflow DAG execution profiling
- Bottleneck detection and analysis
- SQL query optimization suggestions
- Spark job tuning recommendations
- Cost analysis and optimization
- Historical performance trending

**Usage:**
```bash
# Analyze Airflow DAG
python scripts/etl_performance_optimizer.py \
    --airflow-db postgresql://host/airflow \
    --dag-id sales_etl_pipeline \
    --days 30 \
    --optimize

# Spark job analysis
python scripts/etl_performance_optimizer.py \
    --spark-history-server http://spark-history:18080 \
    --app-id app-20250115-001
```

**Complete Documentation:** See [tools.md](references/tools.md) for profiling options, optimization strategies, and cost analysis.

## Reference Documentation

### Frameworks ([frameworks.md](references/frameworks.md))

Comprehensive data engineering frameworks and patterns:
- **Architecture Patterns:** Lambda, Kappa, Medallion, Microservices data architecture
- **Data Modeling:** Dimensional (Kimball), Data Vault 2.0, One Big Table
- **ETL/ELT Patterns:** Full load, incremental load, CDC, SCD, idempotent pipelines
- **Data Quality:** Complete framework covering all quality dimensions
- **DataOps:** CI/CD for data pipelines, testing strategies, monitoring
- **Orchestration:** Airflow DAG patterns, backfill strategies
- **Streaming:** Kafka-based streaming, Spark Structured Streaming
- **Governance:** Data catalog, lineage tracking, access control

### Templates ([templates.md](references/templates.md))

Production-ready code templates and examples:
- **Airflow DAGs:** Complete ETL DAG, incremental load, dynamic task generation
- **Spark Jobs:** Batch processing, streaming, optimized configurations
- **dbt Models:** Staging, intermediate, fact tables, dimensions with SCD Type 2
- **SQL Patterns:** Incremental merge (upsert), deduplication, date spine, window functions
- **Python Pipelines:** Data quality validation class, retry decorators, error handling
- **Docker:** Dockerfiles for data pipelines, Docker Compose for local development
- **Configuration:** dbt project config, Spark configuration, Airflow variables
- **Testing:** pytest fixtures, integration tests, data quality tests

### Tools ([tools.md](references/tools.md))

Python automation tool documentation:
- **pipeline_orchestrator.py:** Complete usage guide, configuration format, DAG templates
- **data_quality_validator.py:** Validation rules, dimension checks, Great Expectations integration
- **etl_performance_optimizer.py:** Performance analysis, query optimization, Spark tuning
- **Integration Patterns:** Airflow, dbt, CI/CD, monitoring systems
- **Best Practices:** Configuration management, error handling, performance, monitoring

## Tech Stack

**Core Technologies:**
- **Languages:** Python 3.8+, SQL, Scala (Spark)
- **Orchestration:** Apache Airflow, Prefect, Dagster
- **Processing:** Apache Spark, dbt, Pandas
- **Streaming:** Apache Kafka, Spark Streaming, Flink
- **Storage:** PostgreSQL, BigQuery, Snowflake, Redshift, S3, GCS
- **Containerization:** Docker, Kubernetes
- **Monitoring:** Datadog, Prometheus, Grafana

**Data Platforms:**
- **Cloud Data Warehouses:** Snowflake, BigQuery, Redshift
- **Data Lakes:** Delta Lake, Apache Iceberg, Apache Hudi
- **Streaming:** Kafka, Kinesis, Pub/Sub
- **Workflow:** Airflow, Prefect, Dagster

## Integration Points

This skill integrates with:
- **Orchestration:** Airflow, Prefect, Dagster for workflow management
- **Transformation:** dbt for SQL transformations and testing
- **Quality:** Great Expectations for data validation
- **Monitoring:** Datadog, Prometheus for pipeline monitoring
- **BI Tools:** Looker, Tableau, Power BI for analytics
- **ML Platforms:** MLflow, Kubeflow for ML pipeline integration
- **Version Control:** Git for pipeline code and configuration

See [tools.md](references/tools.md) for detailed integration patterns and examples.

## Best Practices

**Pipeline Design:**
1. Idempotent operations for safe reruns
2. Incremental processing where possible
3. Clear data lineage and documentation
4. Comprehensive error handling
5. Automated recovery mechanisms

**Data Quality:**
1. Define quality rules early
2. Validate at every pipeline stage
3. Automate quality monitoring
4. Track quality trends over time
5. Block bad data from downstream

**Performance:**
1. Partition large tables by date/region
2. Use columnar formats (Parquet, ORC)
3. Leverage predicate pushdown
4. Optimize for your query patterns
5. Monitor and tune regularly

**Operations:**
1. Version control everything
2. Automate testing and deployment
3. Implement comprehensive monitoring
4. Document runbooks for incidents
5. Regular performance reviews

## Performance Targets

**Pipeline Execution:**
- P50 latency: < 5 minutes (hourly pipelines)
- P95 latency: < 15 minutes
- Success rate: > 99%
- Data freshness: < 1 hour behind source

**Data Quality:**
- Quality score: > 95%
- Completeness: > 99%
- Timeliness: < 2 hours data lag
- Zero critical failures

**Cost Efficiency:**
- Cost per GB processed: < $0.10
- Cloud cost trend: Stable or decreasing
- Resource utilization: > 70%

## Resources

- **Frameworks Guide:** [references/frameworks.md](references/frameworks.md)
- **Code Templates:** [references/templates.md](references/templates.md)
- **Tool Documentation:** [references/tools.md](references/tools.md)
- **Python Scripts:** `scripts/` directory

---

**Version:** 1.0.0
**Last Updated:** 2025-11-08
**Documentation Structure:** Progressive disclosure with comprehensive references