# Data Engineer

> Use when building data pipelines, ETL/ELT workflows, data warehouse design, streaming data infrastructure, or data platform engineering. Trigger phrases: "data pipeline", "ETL", "ELT", "data warehouse", "dbt model", "Airflow DAG", "Spark job", "Kafka stream", "data ingestion", "Snowflake", "BigQuery", "Redshift", "data lake", "data platform", "schema design", "data modeling", "data quality", "data orchestration".

- Skill: `barastrong/data-engineer` (Agent Skill, multi-file: 5 files)
- Install (CLI): `npx skillmds@latest add barastrong/data-engineer`
- Raw SKILL.md: https://api.skillmd.com/api/skills/barastrong/data-engineer/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: barastrong (https://skillmd.com/u/barastrong)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/barastrong/data-engineer

---


# Data Engineer

## Role Summary

A Data Engineer designs, builds, and maintains data infrastructure — pipelines,
warehouses, lakes, and streaming systems that make data reliably available to
analysts, scientists, and applications at scale.

## Core Responsibilities

- Design and build ETL/ELT pipelines (batch and streaming)
- Model data for analytical workloads (dimensional, Data Vault, OBT)
- Implement data quality checks and observability
- Manage data warehouse/lake infrastructure (Snowflake, BigQuery, S3)
- Orchestrate workflows (Airflow, Prefect, Dagster)
- Build streaming data systems (Kafka, Kinesis, Flink)
- Optimize query performance and storage costs
- Collaborate with Data Analysts and Scientists on data model design

## Standard Workflow

1. **Requirements** — understand consumers (analysts, ML models, dashboards),
   latency SLA, data freshness, and volume.
2. **Source Profiling** — understand source schemas, data quality, change
   patterns (CDC, append-only, snapshot).
3. **Design** — choose ingestion pattern, staging strategy, data model
   (dimensional vs. wide table vs. Data Vault).
4. **Implementation** — build ingestion, transformation (dbt or Spark),
   orchestration DAG.
5. **Data Quality** — add tests (not null, unique, referential integrity,
   value ranges), anomaly detection.
6. **Monitoring** — pipeline success/failure alerts, row count checks,
   SLA breach alerts.
7. **Documentation** — data catalog entries, lineage, column descriptions.

## Technology Stack

| Layer | Tools |
|-------|-------|
| Ingestion | Fivetran, Airbyte, Stitch, custom Python/Spark |
| Transformation | dbt (core/cloud), Apache Spark, PySpark, SQL |
| Orchestration | Apache Airflow, Prefect, Dagster, Mage |
| Streaming | Apache Kafka, Kinesis, Flink, Spark Streaming, Pub/Sub |
| Warehouses | Snowflake, BigQuery, Redshift, Databricks, DuckDB |
| Data Lake | S3, GCS, Azure ADLS, Delta Lake, Apache Iceberg, Hudi |
| Quality | Great Expectations, dbt tests, Soda, Monte Carlo |
| Catalog | DataHub, Amundsen, dbt docs, Collibra |

## Best Practices

- ELT over ETL when warehouse compute is cheap (load raw, transform in-warehouse).
- Immutable raw layer — never overwrite source data.
- Idempotent pipelines — re-running produces same result.
- Partitioning on datetime columns for large tables.
- Incremental models over full refresh where possible (cost + time).
- Data contracts between producers and consumers.
- Test in CI: dbt test, data quality assertions before deployment.
- Column-level lineage documentation.

## Anti-Patterns to Avoid

- Mutable landing zone — always append or snapshot raw data.
- Missing data quality tests — silent corrupt data downstream.
- Over-normalization for analytical workloads (OLAP prefers wide tables).
- No partitioning on large fact tables.
- Undocumented transformations — tribal knowledge.
- Hardcoded credentials in DAG code.

## References

- `references/data-modeling-patterns.md` — dimensional, Data Vault, OBT patterns
- `references/pipeline-testing.md` — data quality testing strategies
- `references/streaming-patterns.md` — Kafka and streaming design patterns

## Expected Output Format

1. Pipeline code (dbt models, DAG, Spark job) with tests
2. Data model documentation (schema, grain, business rules)
3. Monitoring/alerting configuration
4. Data catalog entries for new datasets