Senior Data Engineer

Senior Data Engineer

Core Capabilities

• [Capability 1] - [Description]
• [Capability 2] - [Description]
• [Capability 3] - [Description]
• [Capability 4] - [Description]

Key Workflows

Workflow 1: [Workflow Name]

Time: [Duration estimate]

Steps:

1. [Step 1]
2. [Step 2]
3. [Step 3]

Expected Output: [What success looks like]

Workflow 2: [Workflow Name]

Time: [Duration estimate]

Steps:

1. [Step 1]
2. [Step 2]
3. [Step 3]

Expected Output: [What success looks like]

World-class data engineering for production-grade data systems, scalable pipelines, and enterprise data platforms.

Overview

This skill provides comprehensive expertise in data engineering fundamentals through advanced production patterns. From designing medallion architectures to implementing real-time streaming pipelines, it covers the full spectrum of modern data engineering including ETL/ELT design, data quality frameworks, pipeline orchestration, and DataOps practices.

What This Skill Provides:

• Production-ready pipeline templates (Airflow, Spark, dbt)
• Comprehensive data quality validation framework
• Performance optimization and cost analysis tools
• Data architecture patterns (Lambda, Kappa, Medallion)
• Complete DataOps CI/CD workflows

Best For:

• Building scalable data pipelines for enterprise systems
• Implementing data quality and governance frameworks
• Optimizing ETL performance and cloud costs
• Designing modern data architectures (lake, warehouse, lakehouse)
• Production ML/AI data infrastructure

Quick Start

Pipeline Orchestration

# Generate Airflow DAG from configuration
python scripts/pipeline_orchestrator.py --config pipeline_config.yaml --output dags/

# Validate pipeline configuration
python scripts/pipeline_orchestrator.py --config pipeline_config.yaml --validate

# Use incremental load template
python scripts/pipeline_orchestrator.py --template incremental --output dags/

Data Quality Validation

# Validate CSV file with quality checks
python scripts/data_quality_validator.py --input data/sales.csv --output report.html

# Validate database table with custom rules
python scripts/data_quality_validator.py \
    --connection postgresql://user:pass@host/db \
    --table sales_transactions \
    --rules rules/sales_validation.yaml \
    --threshold 0.95

Performance Optimization

# Analyze pipeline performance and get recommendations
python scripts/etl_performance_optimizer.py \
    --airflow-db postgresql://host/airflow \
    --dag-id sales_etl_pipeline \
    --days 30 \
    --optimize

# Analyze Spark job performance
python scripts/etl_performance_optimizer.py \
    --spark-history-server http://spark-history:18080 \
    --app-id app-20250115-001

Core Workflows

1. Building Production Data Pipelines

Steps:

1. Design Architecture: Choose pattern (Lambda, Kappa, Medallion) based on requirements
2. Configure Pipeline: Create YAML configuration with sources, transformations, targets
3. Generate DAG: python scripts/pipeline_orchestrator.py --config config.yaml
4. Add Quality Checks: Define validation rules for data quality
5. Deploy & Monitor: Deploy to Airflow, configure alerts, track metrics

Pipeline Patterns: See frameworks.md for Lambda Architecture, Kappa Architecture, Medallion Architecture (Bronze/Silver/Gold), and Microservices Data patterns.

Templates: See templates.md for complete Airflow DAG templates, Spark job templates, dbt models, and Docker configurations.

2. Data Quality Management

Steps:

1. Define Rules: Create validation rules covering completeness, accuracy, consistency
2. Run Validation: python scripts/data_quality_validator.py --rules rules.yaml
3. Review Results: Analyze quality scores and failed checks
4. Integrate CI/CD: Add validation to pipeline deployment process
5. Monitor Trends: Track quality scores over time

Quality Framework: See frameworks.md for complete Data Quality Framework covering all dimensions (completeness, accuracy, consistency, timeliness, validity).

Validation Templates: See templates.md for validation configuration examples and Python API usage.

3. Data Modeling & Transformation

Steps:

1. Choose Modeling Approach: Dimensional (Kimball), Data Vault 2.0, or One Big Table
2. Design Schema: Define fact tables, dimensions, and relationships
3. Implement with dbt: Create staging, intermediate, and mart models
4. Handle SCD: Implement slowly changing dimension logic (Type 1/2/3)
5. Test & Deploy: Run dbt tests, generate documentation, deploy

Modeling Patterns: See frameworks.md for Dimensional Modeling (Kimball), Data Vault 2.0, One Big Table (OBT), and SCD implementations.

dbt Templates: See templates.md for complete dbt model templates including staging, intermediate, fact tables, and SCD Type 2 logic.

4. Performance Optimization

Steps:

1. Profile Pipeline: Run performance analyzer on recent pipeline executions
2. Identify Bottlenecks: Review execution time breakdown and slow tasks
3. Apply Optimizations: Implement recommendations (partitioning, indexing, batching)
4. Tune Spark Jobs: Optimize memory, parallelism, and shuffle settings
5. Measure Impact: Compare before/after metrics, track cost savings

Optimization Strategies: See frameworks.md for performance best practices including partitioning strategies, query optimization, and Spark tuning.

Analysis Tools: See tools.md for complete documentation on etl_performance_optimizer.py with query analysis and Spark tuning.

Python Tools

pipeline_orchestrator.py

Automated Airflow DAG generation with intelligent dependency resolution and monitoring.

Key Features:

• Generate production-ready DAGs from YAML configuration
• Automatic task dependency resolution
• Built-in retry logic and error handling
• Multi-source support (PostgreSQL, S3, BigQuery, Snowflake)
• Integrated quality checks and alerting

Usage:

# Basic DAG generation
python scripts/pipeline_orchestrator.py --config pipeline_config.yaml --output dags/

# With validation
python scripts/pipeline_orchestrator.py --config config.yaml --validate

# From template
python scripts/pipeline_orchestrator.py --template incremental --output dags/

Complete Documentation: See tools.md for full configuration options, templates, and integration examples.

data_quality_validator.py

Comprehensive data quality validation framework with automated checks and reporting.

Capabilities:

• Multi-dimensional validation (completeness, accuracy, consistency, timeliness, validity)
• Great Expectations integration
• Custom business rule validation
• HTML/PDF report generation
• Anomaly detection
• Historical trend tracking

Usage:

# Validate with custom rules
python scripts/data_quality_validator.py \
    --input data/sales.csv \
    --rules rules/sales_validation.yaml \
    --output report.html

# Database table validation
python scripts/data_quality_validator.py \
    --connection postgresql://host/db \
    --table sales_transactions \
    --threshold 0.95

Complete Documentation: See tools.md for rule configuration, API usage, and integration patterns.

etl_performance_optimizer.py

Pipeline performance analysis with actionable optimization recommendations.

Capabilities:

• Airflow DAG execution profiling
• Bottleneck detection and analysis
• SQL query optimization suggestions
• Spark job tuning recommendations
• Cost analysis and optimization
• Historical performance trending

Usage:

# Analyze Airflow DAG
python scripts/etl_performance_optimizer.py \
    --airflow-db postgresql://host/airflow \
    --dag-id sales_etl_pipeline \
    --days 30 \
    --optimize

# Spark job analysis
python scripts/etl_performance_optimizer.py \
    --spark-history-server http://spark-history:18080 \
    --app-id app-20250115-001

Complete Documentation: See tools.md for profiling options, optimization strategies, and cost analysis.

Reference Documentation

Frameworks (frameworks.md)

Comprehensive data engineering frameworks and patterns:

• Architecture Patterns: Lambda, Kappa, Medallion, Microservices data architecture
• Data Modeling: Dimensional (Kimball), Data Vault 2.0, One Big Table
• ETL/ELT Patterns: Full load, incremental load, CDC, SCD, idempotent pipelines
• Data Quality: Complete framework covering all quality dimensions
• DataOps: CI/CD for data pipelines, testing strategies, monitoring
• Orchestration: Airflow DAG patterns, backfill strategies
• Streaming: Kafka-based streaming, Spark Structured Streaming
• Governance: Data catalog, lineage tracking, access control

Templates (templates.md)

Production-ready code templates and examples:

• Airflow DAGs: Complete ETL DAG, incremental load, dynamic task generation
• Spark Jobs: Batch processing, streaming, optimized configurations
• dbt Models: Staging, intermediate, fact tables, dimensions with SCD Type 2
• SQL Patterns: Incremental merge (upsert), deduplication, date spine, window functions
• Python Pipelines: Data quality validation class, retry decorators, error handling
• Docker: Dockerfiles for data pipelines, Docker Compose for local development
• Configuration: dbt project config, Spark configuration, Airflow variables
• Testing: pytest fixtures, integration tests, data quality tests

Tools (tools.md)

Python automation tool documentation:

• pipeline_orchestrator.py: Complete usage guide, configuration format, DAG templates
• data_quality_validator.py: Validation rules, dimension checks, Great Expectations integration
• etl_performance_optimizer.py: Performance analysis, query optimization, Spark tuning
• Integration Patterns: Airflow, dbt, CI/CD, monitoring systems
• Best Practices: Configuration management, error handling, performance, monitoring

Tech Stack

Core Technologies:

• Languages: Python 3.8+, SQL, Scala (Spark)
• Orchestration: Apache Airflow, Prefect, Dagster
• Processing: Apache Spark, dbt, Pandas
• Streaming: Apache Kafka, Spark Streaming, Flink
• Storage: PostgreSQL, BigQuery, Snowflake, Redshift, S3, GCS
• Containerization: Docker, Kubernetes
• Monitoring: Datadog, Prometheus, Grafana

Data Platforms:

• Cloud Data Warehouses: Snowflake, BigQuery, Redshift
• Data Lakes: Delta Lake, Apache Iceberg, Apache Hudi
• Streaming: Kafka, Kinesis, Pub/Sub
• Workflow: Airflow, Prefect, Dagster

Integration Points

This skill integrates with:

• Orchestration: Airflow, Prefect, Dagster for workflow management
• Transformation: dbt for SQL transformations and testing
• Quality: Great Expectations for data validation
• Monitoring: Datadog, Prometheus for pipeline monitoring
• BI Tools: Looker, Tableau, Power BI for analytics
• ML Platforms: MLflow, Kubeflow for ML pipeline integration
• Version Control: Git for pipeline code and configuration

See tools.md for detailed integration patterns and examples.

Best Practices

Pipeline Design:

1. Idempotent operations for safe reruns
2. Incremental processing where possible
3. Clear data lineage and documentation
4. Comprehensive error handling
5. Automated recovery mechanisms

Data Quality:

1. Define quality rules early
2. Validate at every pipeline stage
3. Automate quality monitoring
4. Track quality trends over time
5. Block bad data from downstream

Performance:

1. Partition large tables by date/region
2. Use columnar formats (Parquet, ORC)
3. Leverage predicate pushdown
4. Optimize for your query patterns
5. Monitor and tune regularly

Operations:

1. Version control everything
2. Automate testing and deployment
3. Implement comprehensive monitoring
4. Document runbooks for incidents
5. Regular performance reviews

Performance Targets

Pipeline Execution:

• P50 latency: < 5 minutes (hourly pipelines)
• P95 latency: < 15 minutes
• Success rate: > 99%
• Data freshness: < 1 hour behind source

Data Quality:

• Quality score: > 95%
• Completeness: > 99%
• Timeliness: < 2 hours data lag
• Zero critical failures

Cost Efficiency:

• Cost per GB processed: < $0.10
• Cloud cost trend: Stable or decreasing
• Resource utilization: > 70%

Resources

• Frameworks Guide: references/frameworks.md
• Code Templates: references/templates.md
• Tool Documentation: references/tools.md
• Python Scripts: scripts/ directory

Version: 1.0.0 Last Updated: 2025-11-08 Documentation Structure: Progressive disclosure with comprehensive references