# Pipeline Architecture Patterns

> Data pipeline architecture patterns for ETL/ELT design, orchestration, and data quality frameworks Use when this capability is needed.

- Skill: `tomevault-io/pipeline-architecture-patterns` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add tomevault-io/pipeline-architecture-patterns`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/pipeline-architecture-patterns/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: tomevault-io (https://skillmd.com/u/tomevault-io)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tomevault-io/pipeline-architecture-patterns

---


# Data Pipeline Architecture Patterns

## Pipeline Architectures

### ETL vs ELT (CRITICAL)
- **ETL**: Extract → Transform (staging) → Load
  - Traditional, on-premise data warehouses
  - Pre-aggregation, complex transformations
- **ELT**: Extract → Load (raw) → Transform (in warehouse)
  - Cloud warehouses (Snowflake, BigQuery)
  - Leverage warehouse compute power

### Lambda Architecture
- Batch layer: historical data processing
- Speed layer: real-time stream processing
- Serving layer: merge batch + real-time views
- Complexity: maintain two codebases

### Kappa Architecture
- Stream-only processing
- Single codebase for batch + real-time
- Reprocessing via replay
- Simpler than Lambda

### Medallion Architecture
- **Bronze**: Raw data (append-only)
- **Silver**: Cleaned, conformed data
- **Gold**: Business-level aggregations
- Databricks pattern

## Orchestration Patterns

### DAG-Based Orchestration
- Airflow, Prefect, Dagster
- Task dependencies as DAG
- Retries, backfills, scheduling

### Event-Driven Orchestration
- Kafka, Pub/Sub triggers
- Real-time, low-latency
- Decoupled producers/consumers

### Hybrid Orchestration
- Scheduled batch + event-driven streams
- Example: Airflow DAG triggered by Kafka event

## Data Quality Frameworks

### Data Contracts (CRITICAL)
- Define schema, freshness, volume expectations
- Producer-consumer agreement
- Break build on violation

### Validation Frameworks
- **Great Expectations**: Python-based expectations
- **dbt tests**: SQL-based tests
- **Soda**: YAML-based checks

### Data Lineage
- Track data origin and transformations
- Debug data quality issues
- Compliance and auditing

## Idempotency Patterns

### Idempotent Design (CRITICAL)
- Same input → same output (no side effects)
- Upserts instead of inserts
- Partition replacement instead of append

### Deduplication
- Use unique keys
- Window-based deduplication
- Consumer group offset management

## References
- [Data Engineering Design Patterns](https://www.oreilly.com/library/view/data-engineering-design/9781098130725/)

---
> Converted and distributed by [TomeVault](https://tomevault.io/claim/baekenough) — claim your Tome and manage your conversions.
<!-- tomevault:4.0:skill_md:2026-04-12 -->