# Stream Processing

> Skill — Stream Processing

- Skill: `sairam0424/stream-processing` (Agent Skill)
- Install (CLI): `npx skillmds@latest add sairam0424/stream-processing`
- Raw SKILL.md: https://api.skillmd.com/api/skills/sairam0424/stream-processing/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: sairam0424 (https://skillmd.com/u/sairam0424)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/sairam0424/stream-processing

---


# Skill — Stream Processing

## When this skill activates
This skill activates when designing real-time data pipelines, implementing event streaming architectures, or building low-latency aggregation systems. Use when data must be processed continuously as it arrives rather than in batch intervals.

## Mandatory actions when this skill is active

### Before writing any code
1. Define processing semantics requirements: at-most-once (fast, lossy), at-least-once (duplicates possible), exactly-once (consistency critical)
2. Select windowing strategy based on use case: tumbling (fixed non-overlapping), sliding (overlapping intervals), session (activity-based gaps)
3. Design watermark strategy for handling late events: fixed delay, percentile-based, or custom heuristic with acceptable lateness window
4. Establish state management approach: in-memory (fast but limited), RocksDB (scalable), remote store (shared state) with checkpointing frequency

### During implementation
- Implement event time processing using timestamps from event payload rather than processing time to handle out-of-order events correctly
- Configure watermarks with balance between latency and completeness: too aggressive drops late events, too conservative increases latency
- Design stateful operations with appropriate state backend and checkpointing: every 5-60 seconds based on throughput and failure recovery requirements
- Build stream joins with care for state size: time-bounded joins, interval joins, or temporal table joins to prevent unbounded state growth
- Implement exactly-once semantics using transactional producers, idempotent consumers, and two-phase commit when required
- Handle late events with configurable strategy: drop and log, emit to side output, update previous window results with allowed lateness
- Design backpressure handling: flow control to slow down producers, buffer sizing, and scaling policies for consumer parallelism

### After implementation
- Monitor stream processing lag metrics: consumer group lag, watermark delay, late events dropped, and checkpoint duration
- Build observability dashboards: throughput (events/sec), latency percentiles (p50/p95/p99), error rates, and state size growth
- Create alerting on critical conditions: lag exceeding SLA, checkpoint failures, repeated consumer rebalancing, state size approaching limits
- Document failure recovery procedures: checkpoint restoration, consumer group reset, and data replay strategies

## Self-check before task completion
- [ ] Processing semantics (at-least-once or exactly-once) verified with end-to-end tests including failure injection
- [ ] Watermark strategy handles late events appropriately with acceptable data loss or latency trade-off documented
- [ ] State management configured with checkpointing and tested with recovery from checkpoint after failure
- [ ] Stream joins bounded in time with monitoring for state size growth and cleanup of expired state
- [ ] Backpressure scenarios tested with producer slowdown and consumer scaling validated under load