# Dataflow Pipeline

> Dataflow Pipeline

- Skill: `kienbui1995/dataflow-pipeline` (Agent Skill)
- Install (CLI): `npx skillmds@latest add kienbui1995/dataflow-pipeline`
- Raw SKILL.md: https://api.skillmd.com/api/skills/kienbui1995/dataflow-pipeline/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: kienbui1995 (https://skillmd.com/u/kienbui1995)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/kienbui1995/dataflow-pipeline

---


# Dataflow Pipeline

## When to Use
- Building ETL pipelines (batch or streaming) on GCP
- Choosing between Dataflow and Dataproc for a workload
- Designing windowing or late-data handling for streaming
- Preparing for GCP Professional Data Engineer exam (highest weight domain)

## Core Jobs

### 1. Dataflow vs Dataproc Decision
| Factor | Choose Dataflow | Choose Dataproc |
|--------|----------------|-----------------|
| Runtime | Apache Beam pipelines | Spark/Hadoop ecosystem |
| Management | Fully managed, serverless | Cluster to manage (or autoscaling) |
| Streaming | Native (Pub/Sub → BQ) | Spark Streaming (more complex) |
| Existing code | Greenfield | Migrating existing Spark jobs |
| Cost model | Per vCPU/memory/hour | Cluster uptime |

### 2. Pipeline Design (Apache Beam)
Core abstractions:
- **PCollection** — distributed dataset (bounded for batch, unbounded for streaming)
- **PTransform** — operation on a PCollection (Map, Filter, GroupByKey, Combine)
- **ParDo** — element-wise transformation (like map/flatMap)
- **Pipeline** — DAG of PTransforms applied to PCollections

### 3. Windowing (Streaming)
- **Fixed windows** — equal non-overlapping intervals (e.g., 1-minute buckets)
- **Sliding windows** — overlapping intervals (e.g., 10-min window every 1 min)
- **Session windows** — gap-based; group events within a user session
- **Global window** — default; all elements in one window (use with triggers for streaming)

### 4. Watermarks and Late Data
- **Watermark** — Dataflow's estimate of how far behind real-time the data is
- Late elements arrive after the watermark passes their window
- Handle with `.withAllowedLateness(Duration.standardMinutes(10))`
- Late data triggers go to a dead-letter or side output

### 5. Triggers
- **Default (event time)** — fire when watermark passes window end
- **AfterProcessingTime** — fire after processing-time delay (for low-latency)
- **AfterCount** — fire after N elements accumulated
- **Composite triggers** — combine with `.orFinally()`, `.repeatedly()`

### 6. Templates
- **Classic Templates** — staged as GCS files; no runtime parameters
- **Flex Templates** — packaged as Docker images; support runtime parameters; preferred

## Key Concepts
- **Fusion** — Dataflow optimization: merges compatible transforms to reduce shuffles
- **Worker autoscaling** — Dataflow scales workers based on backlog
- **Shuffle service** — offloads GroupByKey shuffle to Dataflow backend (reduces worker cost)
- **Streaming Engine** — offloads windowing/state to backend; reduces memory on workers

## Checklist
- [ ] Use Flex Templates (not Classic) for new pipelines?
- [ ] Late data handled with `.withAllowedLateness()`?
- [ ] Side outputs used for dead-letter / error records?
- [ ] GroupByKey minimized (use Combine where possible)?
- [ ] Dataflow Shuffle service enabled for batch jobs?
- [ ] Streaming Engine enabled for streaming jobs?
- [ ] Pipeline tested locally with DirectRunner before deploying?

## Output Format
- 🔴 **Critical** — unbounded PCollection without windowing in streaming pipeline
- 🟡 **Warning** — Classic Template used (prefer Flex), no late data handling
- 🟢 **Suggestion** — Shuffle service / Streaming Engine not enabled

## Exam Tips
- **Watermark** = when Dataflow thinks all data with that timestamp has arrived
- Late data arrives AFTER the watermark → use `.withAllowedLateness()` to capture it
- **Dataflow → BigQuery streaming inserts** = standard pattern for real-time analytics
- **Dataproc** = use for existing Spark/Hadoop code migration, not greenfield
- Flex Templates > Classic Templates for all new pipelines (runtime params, easier updates)
- `DirectRunner` = local testing; `DataflowRunner` = GCP execution