Kafka

Apache Kafka Skill

Distributed event streaming platform.

Ecosystem Graph Preview

graph LR
  kafka["kafka"]:::core
  classDef core fill:#f9f,stroke:#333,stroke-width:4px;
  kafka -- "alternative to" --> rabbitmq
  kafka -- "alternative to" --> redis-streams
  nats -- "related to" --> kafka
  rabbitmq -- "alternative to" --> kafka
  redis-streams -- "alternative to" --> kafka

Recommended Next Skills

• rabbitmq (Score: 0.9) Why: Direct relationship, Both are Messaging, Shared ecosystem (infrastructure), Can deploy to aws
• redis-streams (Score: 0.8) Why: Direct relationship, Both are Messaging, Can deploy to aws
• nats (Score: 0.7) Why: Direct relationship, Both are Messaging

Quick Start

Kafka is not a queue; it is a distributed, append-only log. It excels at handling massive event streams (like clickstreams or IoT telemetry) by distributing partitions across a cluster.

docker run -p 9092:9092 apache/kafka:latest

Production Patterns

Consumer Groups and Partitions

Kafka guarantees message ordering only within a single partition. To scale out, you must increase the number of partitions for a topic. The maximum number of concurrent consumers in a group is strictly equal to the number of partitions.

Architecture & Scaling

Log Retention

Unlike RabbitMQ where messages are deleted once consumed, Kafka retains messages on disk based on a retention policy (e.g., 7 days or 100GB). This allows new consumer groups to replay the entire history of events from the beginning.

Error Recovery

If a consumer is too slow, Kafka will trigger a 'rebalance', pausing message processing for the entire group. Ensure your consumer's max.poll.interval.ms is tuned properly, and keep processing logic completely asynchronous from the polling loop.

Security Notes

Enable SASL/SCRAM for client authentication and TLS for in-transit encryption. Kafka without security is completely open to the network.

References

• Kafka Docs

Why use this skill

Use this when your agent works with kafka — structured patterns beat pasted docs and prevent common hallucinations.

AI pitfalls

• Using outdated SDK or API versions from training data
• Inventing environment variable names
• Omitting error handling and retry logic

Production checklist

• Secrets in environment variables, not source code
• Error handling and logging in place
• Rate limits and timeouts configured

Related skills

• rabbitmq — alternative to
• redis-streams — alternative to

Last Verified: 2026-07-02