How do I install the Databricks Spark Structured Streaming skill?

Run npx skillmds@latest add databrickslabs/databricks-spark-structured-streaming in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Databricks Spark Structured Streaming safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Databricks Spark Structured Streaming?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Databricks Spark Structured Streaming free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Databricks Spark Structured Streaming?

databrickslabs (@databrickslabs) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Databricks Spark Structured Streaming by databrickslabs

Name: Databricks Spark Structured Streaming
Author: databrickslabs

Databricks Spark Structured Streaming

Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.

databrickslabs ddb52a0 10 files · 108.1 KB Updated Sep 17, 2026

File contents

Spark Structured Streaming

Production-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.

Quick Start

from pyspark.sql.functions import col, from_json

# Basic Kafka to Delta streaming
df = (spark
    .readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")
    .option("subscribe", "topic")
    .load()
    .select(from_json(col("value").cast("string"), schema).alias("data"))
    .select("data.*")
)

df.writeStream \
    .format("delta") \
    .outputMode("append") \
    .option("checkpointLocation", "/Volumes/catalog/checkpoints/stream") \
    .trigger(processingTime="30 seconds") \
    .start("/delta/target_table")

Core Patterns

Pattern	Description	Reference
Kafka Streaming	Kafka to Delta, Kafka to Kafka, Real-Time Mode	See kafka-streaming.md
Stream Joins	Stream-stream joins, stream-static joins	See stream-stream-joins.md, stream-static-joins.md
Multi-Sink Writes	Write to multiple tables, parallel merges	See multi-sink-writes.md
Merge Operations	MERGE performance, parallel merges, optimizations	See merge-operations.md

Configuration

Topic	Description	Reference
Checkpoints	Checkpoint management and best practices	See checkpoint-best-practices.md
Stateful Operations	Watermarks, state stores, RocksDB configuration	See stateful-operations.md
Trigger & Cost	Trigger selection, cost optimization, RTM	See trigger-and-cost-optimization.md

Best Practices

Topic	Description	Reference
Production Checklist	Comprehensive best practices	See streaming-best-practices.md

Production Checklist

Checkpoint location is persistent (UC volumes, not DBFS)
Unique checkpoint per stream
Fixed-size cluster (no autoscaling for streaming)
Monitoring configured (input rate, lag, batch duration)
Exactly-once verified (txnVersion/txnAppId)
Watermark configured for stateful operations
Left joins for stream-static (not inner)

---name: databricks-spark-structured-streamingdescription: Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.---# Spark Structured StreamingProduction-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.## Quick Start```pythonfrom pyspark.sql.functions import col, from_json# Basic Kafka to Delta streamingdf = (spark .readStream .format("kafka") .option("kafka.bootstrap.servers", "broker:9092") .option("subscribe", "topic") .load() .select(from_json(col("value").cast("string"), schema).alias("data")) .select("data.*"))df.writeStream \ .format("delta") \ .outputMode("append") \ .option("checkpointLocation", "/Volumes/catalog/checkpoints/stream") \ .trigger(processingTime="30 seconds") \ .start("/delta/target_table")```## Core Patterns| Pattern | Description | Reference ||---------|-------------|-----------|| **Kafka Streaming** | Kafka to Delta, Kafka to Kafka, Real-Time Mode | See [kafka-streaming.md](kafka-streaming.md) || **Stream Joins** | Stream-stream joins, stream-static joins | See [stream-stream-joins.md](stream-stream-joins.md), [stream-static-joins.md](stream-static-joins.md) || **Multi-Sink Writes** | Write to multiple tables, parallel merges | See [multi-sink-writes.md](multi-sink-writes.md) || **Merge Operations** | MERGE performance, parallel merges, optimizations | See [merge-operations.md](merge-operations.md) |## Configuration| Topic | Description | Reference ||-------|-------------|-----------|| **Checkpoints** | Checkpoint management and best practices | See [checkpoint-best-practices.md](checkpoint-best-practices.md) || **Stateful Operations** | Watermarks, state stores, RocksDB configuration | See [stateful-operations.md](stateful-operations.md) || **Trigger & Cost** | Trigger selection, cost optimization, RTM | See [trigger-and-cost-optimization.md](trigger-and-cost-optimization.md) |## Best Practices| Topic | Description | Reference ||-------|-------------|-----------|| **Production Checklist** | Comprehensive best practices | See [streaming-best-practices.md](streaming-best-practices.md) |## Production Checklist- [ ] Checkpoint location is persistent (UC volumes, not DBFS)- [ ] Unique checkpoint per stream- [ ] Fixed-size cluster (no autoscaling for streaming)- [ ] Monitoring configured (input rate, lag, batch duration)- [ ] Exactly-once verified (txnVersion/txnAppId)- [ ] Watermark configured for stateful operations- [ ] Left joins for stream-static (not inner)

Frequently asked questions