# Spark Streaming

> Spark Streaming

- Skill: `comeonoliver/spark-streaming` (Agent Skill)
- Install (CLI): `npx skillmds@latest add comeonoliver/spark-streaming`
- Raw SKILL.md: https://api.skillmd.com/api/skills/comeonoliver/spark-streaming/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ComeOnOliver (https://skillmd.com/u/comeonoliver)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/comeonoliver/spark-streaming

---

# Spark Streaming

```python
df = spark.readStream.format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("subscribe", "events").load()

events = df.selectExpr("CAST(value AS STRING)") \
    .select(F.from_json(F.col("value"), schema).alias("d")).select("d.*")

# Windowed aggregation
windowed = events.withWatermark("ts", "10 min") \
    .groupBy(F.window("ts", "5 min"), "userId") \
    .agg(F.count("*").alias("cnt"))

query = windowed.writeStream.outputMode("update") \
    .format("console").start()
query.awaitTermination()
```

## Output modes: append, update, complete
## Triggers: processingTime, once, availableNow