barastrong Data Engineer
Role Summary
A Data Engineer designs, builds, and maintains data infrastructure — pipelines, warehouses, lakes, and streaming systems that make data reliably available to analysts, scientists, and applications at scale.
Core Responsibilities
- Design and build ETL/ELT pipelines (batch and streaming)
- Model data for analytical workloads (dimensional, Data Vault, OBT)
- Implement data quality checks and observability
- Manage data warehouse/lake infrastructure (Snowflake, BigQuery, S3)
- Orchestrate workflows (Airflow, Prefect, Dagster)
- Build streaming data systems (Kafka, Kinesis, Flink)
- Optimize query performance and storage costs
- Collaborate with Data Analysts and Scientists on data model design
Standard Workflow
- Requirements — understand consumers (analysts, ML models, dashboards), latency SLA, data freshness, and volume.
- Source Profiling — understand source schemas, data quality, change patterns (CDC, append-only, snapshot).
- Design — choose ingestion pattern, staging strategy, data model (dimensional vs. wide table vs. Data Vault).
- Implementation — build ingestion, transformation (dbt or Spark), orchestration DAG.
- Data Quality — add tests (not null, unique, referential integrity, value ranges), anomaly detection.
- Monitoring — pipeline success/failure alerts, row count checks, SLA breach alerts.
- Documentation — data catalog entries, lineage, column descriptions.
Technology Stack
| Layer | Tools |
|---|---|
| Ingestion | Fivetran, Airbyte, Stitch, custom Python/Spark |
| Transformation | dbt (core/cloud), Apache Spark, PySpark, SQL |
| Orchestration | Apache Airflow, Prefect, Dagster, Mage |
| Streaming | Apache Kafka, Kinesis, Flink, Spark Streaming, Pub/Sub |
| Warehouses | Snowflake, BigQuery, Redshift, Databricks, DuckDB |
| Data Lake | S3, GCS, Azure ADLS, Delta Lake, Apache Iceberg, Hudi |
| Quality | Great Expectations, dbt tests, Soda, Monte Carlo |
| Catalog | DataHub, Amundsen, dbt docs, Collibra |
Best Practices
- ELT over ETL when warehouse compute is cheap (load raw, transform in-warehouse).
- Immutable raw layer — never overwrite source data.
- Idempotent pipelines — re-running produces same result.
- Partitioning on datetime columns for large tables.
- Incremental models over full refresh where possible (cost + time).
- Data contracts between producers and consumers.
- Test in CI: dbt test, data quality assertions before deployment.
- Column-level lineage documentation.
Anti-Patterns to Avoid
- Mutable landing zone — always append or snapshot raw data.
- Missing data quality tests — silent corrupt data downstream.
- Over-normalization for analytical workloads (OLAP prefers wide tables).
- No partitioning on large fact tables.
- Undocumented transformations — tribal knowledge.
- Hardcoded credentials in DAG code.
References
references/data-modeling-patterns.md— dimensional, Data Vault, OBT patternsreferences/pipeline-testing.md— data quality testing strategiesreferences/streaming-patterns.md— Kafka and streaming design patterns
Expected Output Format
- Pipeline code (dbt models, DAG, Spark job) with tests
- Data model documentation (schema, grain, business rules)
- Monitoring/alerting configuration
- Data catalog entries for new datasets