Batch Vs Streaming

Batch vs streaming

The deciding question is what the freshest useful result is worth. Batch is simpler to build, test, and rerun; streaming buys latency with a permanent operational tax. Interrogate the latency requirement before paying it.

Method

1. Get the latency requirement in numbers, from the consumer. "Real-time" usually decomposes: fraud checks (sub-second, genuinely streaming), operational dashboards (minutes: micro-batch), executive reporting (hours: batch). Ask what decision changes if the data is 15 minutes old; if the answer is "none", the requirement is batch wearing a costume.
2. Default batch; escalate in steps. Hourly/daily batch (see data-pipeline-design), then frequent micro-batch (5-15 min, same tooling, most of the perceived streaming value), then true streaming (Flink/Kafka Streams-class) only for event-time correctness at low seconds or sub-second reaction. Each step up trades away rerun simplicity.
3. If streaming, adopt the semantics package whole. Event time vs processing time, watermarks for lateness, windows (tumbling, sliding, session) with allowed-lateness policy, and stateful operator checkpointing for effectively-once results (see delivery-guarantees, idempotent-consumers on the sink side). Streaming without watermark discipline produces answers that are fast and quietly wrong.
4. Plan the reprocessing path either way. Batch replays from raw naturally; streaming needs retained log history (see message-queues) plus rewind-and-replay runbooks, or a batch backfill path beside it. A streaming pipeline that cannot reprocess yesterday's bug is append-only regret.
5. Avoid dual codebases when both latencies are needed. Prefer one engine/framework expressing both (unified APIs, streaming with batch backfill mode) or micro-batch covering both requirements, over maintaining separate streaming and batch implementations of the same logic that drift apart (the classic lambda-architecture failure).
6. Budget the operational delta. Streaming means: 24/7 stateful jobs with checkpoint storage, lag monitoring and paging (see backpressure), rebalance/upgrade choreography, and skills on-call. Cost it against the batch alternative explicitly; latency that nobody consumes does not pay this bill.

Boundaries

• CDC replication is streaming infrastructure but not stream processing; moving rows continuously is far simpler than computing on them continuously (see change-data-capture).
• Operational event-driven services (see event-driven-architecture) react to single events; this skill is about analytics computation over flows.
• Latency requirements harden over time; revisit the decision when consumers change, not just when infrastructure fashion does.