sairam0424 Skill — Stream Processing
When this skill activates
This skill activates when designing real-time data pipelines, implementing event streaming architectures, or building low-latency aggregation systems. Use when data must be processed continuously as it arrives rather than in batch intervals.
Mandatory actions when this skill is active
Before writing any code
- Define processing semantics requirements: at-most-once (fast, lossy), at-least-once (duplicates possible), exactly-once (consistency critical)
- Select windowing strategy based on use case: tumbling (fixed non-overlapping), sliding (overlapping intervals), session (activity-based gaps)
- Design watermark strategy for handling late events: fixed delay, percentile-based, or custom heuristic with acceptable lateness window
- Establish state management approach: in-memory (fast but limited), RocksDB (scalable), remote store (shared state) with checkpointing frequency
During implementation
- Implement event time processing using timestamps from event payload rather than processing time to handle out-of-order events correctly
- Configure watermarks with balance between latency and completeness: too aggressive drops late events, too conservative increases latency
- Design stateful operations with appropriate state backend and checkpointing: every 5-60 seconds based on throughput and failure recovery requirements
- Build stream joins with care for state size: time-bounded joins, interval joins, or temporal table joins to prevent unbounded state growth
- Implement exactly-once semantics using transactional producers, idempotent consumers, and two-phase commit when required
- Handle late events with configurable strategy: drop and log, emit to side output, update previous window results with allowed lateness
- Design backpressure handling: flow control to slow down producers, buffer sizing, and scaling policies for consumer parallelism
After implementation
- Monitor stream processing lag metrics: consumer group lag, watermark delay, late events dropped, and checkpoint duration
- Build observability dashboards: throughput (events/sec), latency percentiles (p50/p95/p99), error rates, and state size growth
- Create alerting on critical conditions: lag exceeding SLA, checkpoint failures, repeated consumer rebalancing, state size approaching limits
- Document failure recovery procedures: checkpoint restoration, consumer group reset, and data replay strategies
Self-check before task completion
- Processing semantics (at-least-once or exactly-once) verified with end-to-end tests including failure injection
- Watermark strategy handles late events appropriately with acceptable data loss or latency trade-off documented
- State management configured with checkpointing and tested with recovery from checkpoint after failure
- Stream joins bounded in time with monitoring for state size growth and cleanup of expired state
- Backpressure scenarios tested with producer slowdown and consumer scaling validated under load