Spark Structured Streaming
Production-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.
Quick Start
from pyspark.sql.functions import col, from_json
# Basic Kafka to Delta streaming
df = (spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "broker:9092")
.option("subscribe", "topic")
.load()
.select(from_json(col("value").cast("string"), schema).alias("data"))
.select("data.*")
)
df.writeStream \
.format("delta") \
.outputMode("append") \
.option("checkpointLocation", "/Volumes/catalog/checkpoints/stream") \
.trigger(processingTime="30 seconds") \
.start("/delta/target_table")
Core Patterns
| Pattern |
Description |
Reference |
| Kafka Streaming |
Kafka to Delta, Kafka to Kafka, Real-Time Mode |
See references/kafka-streaming.md |
| Real-Time Mode (RTM) |
Sub-second E2E latency — cluster setup, slot math, supported ops (incl. stream-stream inner join on DBR 18+), transformWithState, observability, error classes, delivery semantics |
See references/real-time-mode.md |
| Lakebase Sink |
Write streaming records into Lakebase Postgres with transactional upserts. Native format("postgresql") sink (DBR 18.3+) and manual foreach sink as a fallback |
See references/lakebase-sink-python.md |
| Stream Joins |
Stream-stream joins, stream-static joins |
See references/stream-stream-joins.md, references/stream-static-joins.md |
| Multi-Sink Writes |
Write to multiple tables, parallel merges |
See references/multi-sink-writes.md |
| Merge Operations |
MERGE performance, parallel merges, optimizations |
See references/merge-operations.md |
Custom State (transformWithState) |
Built-in stateful operator or transformWithState; timers (timeMode="ProcessingTime" never lets AvailableNow jobs finish), TTL, initial state, schema evolution, state reader, migrating from applyInPandasWithState |
See references/transform-with-state.md |
Configuration
| Topic |
Description |
Reference |
| Checkpoints |
Checkpoint management and best practices |
See references/checkpoint-best-practices.md |
| Stateful Operations |
Watermarks, state stores, RocksDB configuration |
See references/stateful-operations.md |
| Trigger & Cost |
Trigger selection, cost optimization, RTM |
See references/trigger-and-cost-optimization.md |
Best Practices
| Topic |
Description |
Reference |
| Production Checklist |
Comprehensive best practices |
See references/streaming-best-practices.md |
Production Checklist
1---2name: databricks-spark-structured-streaming3description: "Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, writing custom stateful logic with transformWithState (timers, TTL, per-key state; preferred over applyInPandasWithState and flatMapGroupsWithState for new code), optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance."4compatibility: Requires databricks CLI (>= v1.0.0)5metadata:6 source: https://skillmd.com/skills/stanfish06/databricks-spark-structured-streaming7 version: "0.1.0"8parent: databricks-core9---1011# Spark Structured Streaming1213Production-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.1415## Quick Start1617```python18from pyspark.sql.functions import col, from_json1920# Basic Kafka to Delta streaming21df = (spark22 .readStream23 .format("kafka")24 .option("kafka.bootstrap.servers", "broker:9092")25 .option("subscribe", "topic")26 .load()27 .select(from_json(col("value").cast("string"), schema).alias("data"))28 .select("data.*")29)3031df.writeStream \32 .format("delta") \33 .outputMode("append") \34 .option("checkpointLocation", "/Volumes/catalog/checkpoints/stream") \35 .trigger(processingTime="30 seconds") \36 .start("/delta/target_table")37```3839## Core Patterns4041| Pattern | Description | Reference |42|---------|-------------|-----------|43| **Kafka Streaming** | Kafka to Delta, Kafka to Kafka, Real-Time Mode | See [references/kafka-streaming.md](references/kafka-streaming.md) |44| **Real-Time Mode (RTM)** | Sub-second E2E latency — cluster setup, slot math, supported ops (incl. stream-stream inner join on DBR 18+), `transformWithState`, observability, error classes, delivery semantics | See [references/real-time-mode.md](references/real-time-mode.md) |45| **Lakebase Sink** | Write streaming records into Lakebase Postgres with transactional upserts. Native `format("postgresql")` sink (DBR 18.3+) and manual `foreach` sink as a fallback | See [references/lakebase-sink-python.md](references/lakebase-sink-python.md) |46| **Stream Joins** | Stream-stream joins, stream-static joins | See [references/stream-stream-joins.md](references/stream-stream-joins.md), [references/stream-static-joins.md](references/stream-static-joins.md) |47| **Multi-Sink Writes** | Write to multiple tables, parallel merges | See [references/multi-sink-writes.md](references/multi-sink-writes.md) |48| **Merge Operations** | MERGE performance, parallel merges, optimizations | See [references/merge-operations.md](references/merge-operations.md) |49| **Custom State (`transformWithState`)** | Built-in stateful operator or `transformWithState`; timers (`timeMode="ProcessingTime"` never lets `AvailableNow` jobs finish), TTL, initial state, schema evolution, state reader, migrating from `applyInPandasWithState` | See [references/transform-with-state.md](references/transform-with-state.md) |5051## Configuration5253| Topic | Description | Reference |54|-------|-------------|-----------|55| **Checkpoints** | Checkpoint management and best practices | See [references/checkpoint-best-practices.md](references/checkpoint-best-practices.md) |56| **Stateful Operations** | Watermarks, state stores, RocksDB configuration | See [references/stateful-operations.md](references/stateful-operations.md) |57| **Trigger & Cost** | Trigger selection, cost optimization, RTM | See [references/trigger-and-cost-optimization.md](references/trigger-and-cost-optimization.md) |5859## Best Practices6061| Topic | Description | Reference |62|-------|-------------|-----------|63| **Production Checklist** | Comprehensive best practices | See [references/streaming-best-practices.md](references/streaming-best-practices.md) |6465## Production Checklist6667- [ ] Checkpoint location is persistent (UC volumes, not DBFS)68- [ ] Unique checkpoint per stream69- [ ] Fixed-size cluster (no autoscaling for streaming)70- [ ] Monitoring configured (input rate, lag, batch duration)71- [ ] Exactly-once verified (txnVersion/txnAppId)72- [ ] Watermark configured for stateful operations73- [ ] New custom state uses `transformWithState` where supported, with `timeMode="EventTime"` or `"None"` in scheduled or serverless jobs74- [ ] Left joins for stream-static (not inner)