Kafka Messaging Skill
A practical playbook for building reliable Kafka producers and consumers, with
first-class retry, dead-letter queue (DLQ), and error handling. Examples
use Spring Kafka, but the patterns (idempotent producers, consumer offset
management, non-blocking retry, poison-message isolation) apply to any client.
Work outside-in: settle configuration first, then build the producer and
consumer, then make them resilient with retry → error handler → DLQ, and
always finish by proving it with an integration test.
When to Use
- Writing a new Kafka producer or consumer
- Making a consumer resilient: retries with backoff, then route failures to a DLQ
- Centralizing consumer error handling (transient vs. non-retryable)
- Configuring brokers, serializers, consumer groups, acks, idempotence
- Writing integration tests for Kafka flows (Testcontainers / EmbeddedKafka)
The Workflow
flowchart LR
CFG[1. Configuration] --> PROD[2. Producer]
CFG --> CONS[3. Consumer]
CONS --> RETRY[4. Retry]
RETRY --> ERR[5. Error Handler]
ERR --> DLQ[6. DLQ]
PROD --> TEST[7. Integration Test]
DLQ --> TEST
1. Configuration
Get the broker, serializers, and consumer-group settings right before writing code.
Decide delivery guarantees up front: acks=all + idempotent producer for no
duplicates/loss; pick auto-offset-reset and disable auto-commit for reliable
consumers.
See Configuration.
2. Producer
Send records with an explicit key (for partition ordering), handle the async send
result, and enable idempotence for exactly-once delivery semantics.
See Producer.
3. Consumer
Consume with @KafkaListener, choose manual vs. auto acknowledgment, and keep
listener methods fast and idempotent (consumers may see a record more than once).
See Consumer.
4. Retry
Retry transient failures only. Prefer non-blocking retry topics
(@RetryableTopic) over blocking the consumer thread; use bounded attempts with
exponential backoff.
See Retry & Backoff.
5. Error Handler
Centralize failure routing in a DefaultErrorHandler: classify exceptions as
retryable vs. non-retryable (fatal), and hand exhausted records to the DLQ recoverer.
See Error Handler.
6. DLQ (Dead-Letter Queue)
Send poison messages to a dead-letter topic instead of blocking the partition.
Preserve the original payload + failure metadata so messages can be inspected and
replayed.
See Dead-Letter Queue.
7. Integration Test
Prove the end-to-end flow — including retry and DLQ routing — with Testcontainers
(real broker) or @EmbeddedKafka (in-memory).
See Integration Testing.
Quick Checklist
Reference Files
1---2name: kafka-messaging3description: Implementation playbook for building production-grade Apache Kafka producers and consumers (Spring Kafka focus). Use when asked to: write a Kafka producer or consumer, add retry/backoff, set up a dead-letter queue (DLQ), handle consumer errors, configure brokers/serializers/consumer groups, or write Kafka integration tests. Triggers: "Kafka", "producer", "consumer", "KafkaTemplate", "@KafkaListener", "retry", "backoff", "dead letter queue", "DLQ", "dead-letter topic", "error handler", "DefaultErrorHandler", "SeekToCurrent", "consumer group", "offset", "serializer", "deserializer", "idempotent producer", "acks", "Testcontainers", "EmbeddedKafka", "Kafka integration test".4---56# Kafka Messaging Skill78A practical playbook for building reliable Kafka producers and consumers, with9first-class **retry**, **dead-letter queue (DLQ)**, and **error handling**. Examples10use **Spring Kafka**, but the patterns (idempotent producers, consumer offset11management, non-blocking retry, poison-message isolation) apply to any client.1213Work outside-in: settle **configuration** first, then build the **producer** and14**consumer**, then make them resilient with **retry → error handler → DLQ**, and15always finish by proving it with an **integration test**.1617---1819## When to Use2021- Writing a new Kafka producer or consumer22- Making a consumer resilient: retries with backoff, then route failures to a DLQ23- Centralizing consumer error handling (transient vs. non-retryable)24- Configuring brokers, serializers, consumer groups, acks, idempotence25- Writing integration tests for Kafka flows (Testcontainers / EmbeddedKafka)2627---2829## The Workflow3031```mermaid32flowchart LR33 CFG[1. Configuration] --> PROD[2. Producer]34 CFG --> CONS[3. Consumer]35 CONS --> RETRY[4. Retry]36 RETRY --> ERR[5. Error Handler]37 ERR --> DLQ[6. DLQ]38 PROD --> TEST[7. Integration Test]39 DLQ --> TEST40```4142---4344### 1. Configuration4546Get the broker, serializers, and consumer-group settings right before writing code.47Decide delivery guarantees up front: `acks=all` + idempotent producer for no48duplicates/loss; pick `auto-offset-reset` and disable auto-commit for reliable49consumers.5051See [Configuration](./references/configuration.md).5253---5455### 2. Producer5657Send records with an explicit key (for partition ordering), handle the async send58result, and enable idempotence for exactly-once *delivery* semantics.5960See [Producer](./references/producer.md).6162---6364### 3. Consumer6566Consume with `@KafkaListener`, choose manual vs. auto acknowledgment, and keep67listener methods fast and idempotent (consumers may see a record more than once).6869See [Consumer](./references/consumer.md).7071---7273### 4. Retry7475Retry **transient** failures only. Prefer non-blocking retry topics76(`@RetryableTopic`) over blocking the consumer thread; use bounded attempts with77exponential backoff.7879See [Retry & Backoff](./references/retry.md).8081---8283### 5. Error Handler8485Centralize failure routing in a `DefaultErrorHandler`: classify exceptions as86retryable vs. non-retryable (fatal), and hand exhausted records to the DLQ recoverer.8788See [Error Handler](./references/error-handler.md).8990---9192### 6. DLQ (Dead-Letter Queue)9394Send poison messages to a dead-letter topic instead of blocking the partition.95Preserve the original payload + failure metadata so messages can be inspected and96replayed.9798See [Dead-Letter Queue](./references/dlq.md).99100---101102### 7. Integration Test103104Prove the end-to-end flow — including retry and DLQ routing — with Testcontainers105(real broker) or `@EmbeddedKafka` (in-memory).106107See [Integration Testing](./references/integration-test.md).108109---110111## Quick Checklist112113- [ ] Delivery guarantees chosen (`acks`, idempotence, `auto-offset-reset`)114- [ ] Manual ack + disabled auto-commit for at-least-once consumers115- [ ] Producer sets a partition key and handles the async send result116- [ ] Listener logic is idempotent (safe under redelivery)117- [ ] Retry is bounded, with backoff, and scoped to transient errors only118- [ ] Non-retryable exceptions classified as fatal (no wasted retries)119- [ ] Exhausted/poison messages routed to a DLQ with failure metadata120- [ ] Integration test covers happy path **and** DLQ routing121122---123124## Reference Files125126- [Configuration](./references/configuration.md)127- [Producer](./references/producer.md)128- [Consumer](./references/consumer.md)129- [Retry & Backoff](./references/retry.md)130- [Error Handler](./references/error-handler.md)131- [Dead-Letter Queue](./references/dlq.md)132- [Integration Testing](./references/integration-test.md)
AmitSharma1994