How do I install the Data Streaming skill?

Run npx skillmds@latest add quantumquirkxyz/data-streaming in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Data Streaming safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Data Streaming?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Data Streaming free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Data Streaming?

quantumquirkxyz (@quantumquirkxyz) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Data Streaming by quantumquirkxyz — Agent Skill

Name: Data Streaming
Author: quantumquirkxyz

File contents

Contract

Input: stream source descriptions, processing requirements, latency/throughput targets.
Output: topology design + event schema + processing rules.
Side effects: may deploy stream resources when executed.
Dependencies: streaming platform access (Kafka / Kinesis / Pub/Sub / Flink).
Stop condition: design saved; rules documented.
Risk: medium — stream errors propagate quickly; requires testing.
Boundary: designs stream; executes only with explicit approval.

Streaming Pipeline Design

Design a streaming pipeline — event producers, stream processors, consumers — with schema, processing rules, and reliability.

Process

1. Source analysis

Event sources: user actions, sensors, logs, transactions, external APIs.
Event rate (events/sec), burst rate, volume (GB/day).
Event schema: fields, types, optional/required, nesting.

Completion criterion: source profile saved.

2. Event schema

Define Avro / Protobuf / JSON Schema.
Versioning: schema registry (Confluent Schema Registry / AWS Glue / GCP Pub/Sub schema).
Backward/forward compatibility rules.

Completion criterion: schema saved; versioning rules defined.

3. Stream topology

Producers: service A, B, C; publish to topic/stream.
Stream: partitioned by key (e.g. user_id, device_id) for parallel processing.
Processors: stream jobs (Flink / Kafka Streams / Spark Streaming / Kinesis Analytics) — filter, aggregate, enrich, transform.
Consumers: service D reads results; may write to database / cache / warehouse.

Completion criterion: topology diagram saved.

4. Processing rules

Windowing: tumbling, sliding, session, custom.
Aggregation: count, sum, average, max, min; with watermarks.
Enrichment: join with reference data (stream or database lookup).
Exactly-once / at-least-once / at-most-once semantics per use case.
Backpressure: when downstream is slow, how to handle (drop old, throttle, buffer with limit).

Completion criterion: rules saved.

5. Reliability

Replication: stream replicas across zones.
Retention: data retention period; archive to storage for long-term.
Monitoring: lag per partition; error rate; throughput; processing latency.
Alert: lag > threshold; error rate > threshold; partition unassigned.

Completion criterion: reliability rules saved.

---name: data-streamingdescription: Design streaming data pipelines — Kafka, Kinesis, Pub/Sub, Flink — with event schemas, stream processing, and real-time analytics.---## Contract- **Input:** stream source descriptions, processing requirements, latency/throughput targets.- **Output:** topology design + event schema + processing rules.- **Side effects:** may deploy stream resources when executed.- **Dependencies:** streaming platform access (Kafka / Kinesis / Pub/Sub / Flink).- **Stop condition:** design saved; rules documented.- **Risk:** medium — stream errors propagate quickly; requires testing.- **Boundary:** designs stream; executes only with explicit approval.# Streaming Pipeline DesignDesign a **streaming pipeline** — event producers, stream processors, consumers — with schema, processing rules, and reliability.## Process### 1. Source analysis- Event sources: user actions, sensors, logs, transactions, external APIs.- Event rate (events/sec), burst rate, volume (GB/day).- Event schema: fields, types, optional/required, nesting.**Completion criterion:** source profile saved.### 2. Event schema- Define Avro / Protobuf / JSON Schema.- Versioning: schema registry (Confluent Schema Registry / AWS Glue / GCP Pub/Sub schema).- Backward/forward compatibility rules.**Completion criterion:** schema saved; versioning rules defined.### 3. Stream topology- **Producers:** service A, B, C; publish to topic/stream.- **Stream:** partitioned by key (e.g. user_id, device_id) for parallel processing.- **Processors:** stream jobs (Flink / Kafka Streams / Spark Streaming / Kinesis Analytics) — filter, aggregate, enrich, transform.- **Consumers:** service D reads results; may write to database / cache / warehouse.**Completion criterion:** topology diagram saved.### 4. Processing rules- Windowing: tumbling, sliding, session, custom.- Aggregation: count, sum, average, max, min; with watermarks.- Enrichment: join with reference data (stream or database lookup).- Exactly-once / at-least-once / at-most-once semantics per use case.- Backpressure: when downstream is slow, how to handle (drop old, throttle, buffer with limit).**Completion criterion:** rules saved.### 5. Reliability- Replication: stream replicas across zones.- Retention: data retention period; archive to storage for long-term.- Monitoring: lag per partition; error rate; throughput; processing latency.- Alert: lag > threshold; error rate > threshold; partition unassigned.**Completion criterion:** reliability rules saved.

Frequently asked questions

Data Streaming