Contract
- Input: stream sources, processing logic, state requirements, delivery guarantees.
- Output: stream topology + windowing design + exactly-once plan.
- Scope: designs stream jobs; execution requires approval.
- Rule: designs stream jobs; execution requires approval.
- Rule: documented standards override defaults; explicit project rules take precedence.
- Rule: if blocked by missing context or dependencies, surface the blocker before proceeding.
Provenance
| Question |
Answer |
| What is the source of truth? |
The user's request, originating spec/issue, and the skill's declared outputs |
| What is in scope? |
Work covered by the skill's acceptance criteria and completion rules |
| What is explicitly out of scope? |
Files, behaviors, and decisions outside the skill's declared boundary |
| Who or what consumes this artifact afterward? |
The next skill in the workflow or the user |
| What evidence proves it is done? |
Completion criteria met, artifact saved, validation passed |
| What risk remains? |
Subjective judgment calls, missing context, or external dependency failures |
Artifact
Emit StreamProcessingArtifact as both:
- JSON:
.agents/skills/platform/artifacts/stream-processing/{identifier}.json
- Markdown view: same filename with
.md extension
Stream Processing
Design stream processing — Flink, Kafka Streams, RisingWave, windowing, state management, and exactly-once semantics.
Process
1. Source and sink analysis
- Sources: Kafka topics, Kinesis streams, Pulsar, IoT hubs.
- Sinks: databases, data lakes, object stores, downstream streams.
- Schema: Avro, Protobuf, JSON, Parquet; schema registry usage.
Completion criterion: source/sink inventory saved.
2. Processing topology
- Stateless: filter, map, flatMap, enrich.
- Stateful: aggregations, joins, pattern detection.
- Branching: fan-out, side outputs, branching streams.
Completion criterion: topology diagram saved (ASCII / Mermaid).
3. Windowing strategies
- Tumbling: fixed-size, non-overlapping windows.
- Sliding: overlapping windows with slide interval.
- Session: activity-based windows with gap timeout.
- Global: all-data window for bounded streams.
Completion criterion: windowing strategy saved with justification.
4. State management
- Keyed state: value state, list state, map state, reduction state.
- State backend: RocksDB (disk), in-memory (fast, limited), tiered.
- State TTL: retention, cleanup, recovery.
- Checkpointing: synchronous vs asynchronous, barrier alignment.
Completion criterion: state design saved.
5. Exactly-once semantics
- Checkpointing: barrier-based, consistent snapshots.
- Two-phase commit: for sinks that support it (Kafka, transactional DBs).
- Idempotent writes: deduplication keys, upserts.
- Recovery: restore from checkpoint, replay from offsets.
Completion criterion: exactly-once plan saved.
6. Fault tolerance and scaling
- Parallelism: source, operator, sink parallelism.
- Scaling: rescaling with savepoints, state migration.
- Backpressure handling: buffering, flow control, monitoring.
Completion criterion: fault tolerance plan saved.
Rules
- Rule: choose the processing framework against throughput, state size, and operational complexity.
- Rule: define windowing, state TTL, and checkpointing before scaling.
- Rule: separate stateless processing from stateful processing so scaling is explicit.
- Rule: require idempotent sinks or deduplication for sensitive downstream systems.
- Rule: monitor backpressure, checkpoint lag, and state size as primary operational signals.
Completion
- the skill's completion criteria are explicitly checked
- the artifact is saved and validated
- any blockers or skipped validations are documented
- the next consumer is identified or the work is handed off
@include .agents/skills/platform/contract-base.xml
1---2name: "stream-processing"3category: "data"4maturity: "stable"5version: "1"6description: "Stream processing (Flink, Kafka Streams, RisingWave, windowing, state, exactly-once)."7license: "MIT"8capabilities:9 - apply stream processing methods within explicit data boundaries10 - analyze stream processing (flink, kafka streams, risingwave, windowing, state, exactly-once) with assumptions, constraints, and failure modes11 - produce and validate a data artifact with evidence, decisions, and next steps12outputs: "stream topology + windowing design + exactly-once plan."13sideEffects: []14dependencies: []15stopCondition: "Topology diagram saved; windowing strategy defined; exactly-once plan documented."16risk: "medium"17trustTier: "3"18maxIterations: "6"19promptVersion: "2.0"20artifactType: "pull-request"21modelTier: "reasoning"22evaluators: ["behavioral", "regression"]23fixturesPath: .agents/skills/platform/fixtures/behavioral/stream-processing.json24diataxis: "how-to"25tags: ["data"]26compatibility: []27approvalRequired: "true"28approvalFor: []29metadata:30 source: https://skillmd.com/skills/quantumquirkxyz/stream-processing31---3233## Contract3435- Input: stream sources, processing logic, state requirements, delivery guarantees.36- Output: stream topology + windowing design + exactly-once plan.37- Scope: designs stream jobs; execution requires approval.38- Rule: designs stream jobs; execution requires approval.39- Rule: documented standards override defaults; explicit project rules take precedence.40- Rule: if blocked by missing context or dependencies, surface the blocker before proceeding.4142## Provenance4344| Question | Answer |45|---|---|46| What is the source of truth? | The user's request, originating spec/issue, and the skill's declared outputs |47| What is in scope? | Work covered by the skill's acceptance criteria and completion rules |48| What is explicitly out of scope? | Files, behaviors, and decisions outside the skill's declared boundary |49| Who or what consumes this artifact afterward? | The next skill in the workflow or the user |50| What evidence proves it is done? | Completion criteria met, artifact saved, validation passed |51| What risk remains? | Subjective judgment calls, missing context, or external dependency failures525354## Artifact5556Emit `StreamProcessingArtifact` as both:57- JSON: `.agents/skills/platform/artifacts/stream-processing/{identifier}.json`58- Markdown view: same filename with `.md` extension596061# 62# 63# Stream Processing6465Design **stream processing** — Flink, Kafka Streams, RisingWave, windowing, state management, and exactly-once semantics.6667## Process6869### 1. Source and sink analysis70- Sources: Kafka topics, Kinesis streams, Pulsar, IoT hubs.71- Sinks: databases, data lakes, object stores, downstream streams.72- Schema: Avro, Protobuf, JSON, Parquet; schema registry usage.7374**Completion criterion:** source/sink inventory saved.7576### 2. Processing topology77- Stateless: filter, map, flatMap, enrich.78- Stateful: aggregations, joins, pattern detection.79- Branching: fan-out, side outputs, branching streams.8081**Completion criterion:** topology diagram saved (ASCII / Mermaid).8283### 3. Windowing strategies84- Tumbling: fixed-size, non-overlapping windows.85- Sliding: overlapping windows with slide interval.86- Session: activity-based windows with gap timeout.87- Global: all-data window for bounded streams.8889**Completion criterion:** windowing strategy saved with justification.9091### 4. State management92- Keyed state: value state, list state, map state, reduction state.93- State backend: RocksDB (disk), in-memory (fast, limited), tiered.94- State TTL: retention, cleanup, recovery.95- Checkpointing: synchronous vs asynchronous, barrier alignment.9697**Completion criterion:** state design saved.9899### 5. Exactly-once semantics100- Checkpointing: barrier-based, consistent snapshots.101- Two-phase commit: for sinks that support it (Kafka, transactional DBs).102- Idempotent writes: deduplication keys, upserts.103- Recovery: restore from checkpoint, replay from offsets.104105**Completion criterion:** exactly-once plan saved.106107### 6. Fault tolerance and scaling108- Parallelism: source, operator, sink parallelism.109- Scaling: rescaling with savepoints, state migration.110- Backpressure handling: buffering, flow control, monitoring.111112**Completion criterion:** fault tolerance plan saved.113114## Rules115116- Rule: choose the processing framework against throughput, state size, and operational complexity.117- Rule: define windowing, state TTL, and checkpointing before scaling.118- Rule: separate stateless processing from stateful processing so scaling is explicit.119- Rule: require idempotent sinks or deduplication for sensitive downstream systems.120- Rule: monitor backpressure, checkpoint lag, and state size as primary operational signals.121122## Completion123124- the skill's completion criteria are explicitly checked125- the artifact is saved and validated126- any blockers or skipped validations are documented127- the next consumer is identified or the work is handed off128---129@include .agents/skills/platform/contract-base.xml