Public Kafka Expert Base

Kafka Knowledge Base

Foundational guidance for building event streaming applications with Apache Kafka.

Source: Adapted from confluentinc/agent-skills and Apache Kafka best practices.

Topics & Partitions

• Design topics around domain events (order.created, payment.completed), not database tables.
• Partition count determines max consumer parallelism — start with 3-6, scale up as needed.
• Partitions are ordered; messages within a partition have guaranteed order.
• Use partition keys for related messages that must be ordered (e.g., user_id for user events).

Consumer Groups

• All consumers in a group share the workload (each partition → one consumer).
• Multiple groups can independently consume the same topic (fan-out).
• Consumer offset tracking: committed offsets vs. current position.
• Use auto.offset.reset=earliest for new groups that need to process all history.

Producer Configuration

Consumer Configuration

Serialization

• Use Avro/Protobuf/JSON Schema with Schema Registry for type safety.
• Never use Java serialization (not cross-language, not forward-compatible).
• StringSerializer/StringDeserializer only for simple text messages.

Exactly-Once Semantics

• Enable enable.idempotence=true on producer (prevents duplicates from retries).
• Use isolation.level=read_committed on consumer (read only committed transactions).
• For transactional processing: beginTransaction() → process → commitTransaction().

Rebalancing

• Triggered when consumers join/leave a group or partitions are reassigned.
• Use cooperative rebalancing (partition.assignment.strategy=CooperativeStickyAssignor).
• Handle CommitFailedException during rebalance — retry after rebalance completes.
• Use ConsumerRebalanceListener for cleanup on partition revocation.

Guardrails

• Always set acks=all and enable.idempotence=true for producers.
• Always use manual offset commits (enable.auto.commit=false).
• Never use Java serialization for Kafka messages.
• Monitor consumer lag (kafka-consumer-groups --describe).