# Kafka Messaging

> Implementation playbook for building production-grade Apache Kafka producers and consumers (Spring Kafka focus). Use when asked to: write a Kafka producer or consumer, add retry/backoff, set up a dead-letter queue (DLQ), handle consumer errors, configure brokers/serializers/consumer groups, or write Kafka integration tests. Triggers: "Kafka", "producer", "consumer", "KafkaTemplate", "@KafkaListener", "retry", "backoff", "dead letter queue", "DLQ", "dead-letter topic", "error handler", "DefaultErrorHandler", "SeekToCurrent", "consumer group", "offset", "serializer", "deserializer", "idempotent producer", "acks", "Testcontainers", "EmbeddedKafka", "Kafka integration test".

- Skill: `amitsharma1994/kafka-messaging` (Agent Skill, multi-file: 8 files)
- Install (CLI): `npx skillmds@latest add amitsharma1994/kafka-messaging`
- Raw SKILL.md: https://api.skillmd.com/api/skills/amitsharma1994/kafka-messaging/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: AmitSharma1994 (https://skillmd.com/u/amitsharma1994)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/amitsharma1994/kafka-messaging

---


# Kafka Messaging Skill

A practical playbook for building reliable Kafka producers and consumers, with
first-class **retry**, **dead-letter queue (DLQ)**, and **error handling**. Examples
use **Spring Kafka**, but the patterns (idempotent producers, consumer offset
management, non-blocking retry, poison-message isolation) apply to any client.

Work outside-in: settle **configuration** first, then build the **producer** and
**consumer**, then make them resilient with **retry → error handler → DLQ**, and
always finish by proving it with an **integration test**.

---

## When to Use

- Writing a new Kafka producer or consumer
- Making a consumer resilient: retries with backoff, then route failures to a DLQ
- Centralizing consumer error handling (transient vs. non-retryable)
- Configuring brokers, serializers, consumer groups, acks, idempotence
- Writing integration tests for Kafka flows (Testcontainers / EmbeddedKafka)

---

## The Workflow

```mermaid
flowchart LR
    CFG[1. Configuration] --> PROD[2. Producer]
    CFG --> CONS[3. Consumer]
    CONS --> RETRY[4. Retry]
    RETRY --> ERR[5. Error Handler]
    ERR --> DLQ[6. DLQ]
    PROD --> TEST[7. Integration Test]
    DLQ --> TEST
```

---

### 1. Configuration

Get the broker, serializers, and consumer-group settings right before writing code.
Decide delivery guarantees up front: `acks=all` + idempotent producer for no
duplicates/loss; pick `auto-offset-reset` and disable auto-commit for reliable
consumers.

See [Configuration](./references/configuration.md).

---

### 2. Producer

Send records with an explicit key (for partition ordering), handle the async send
result, and enable idempotence for exactly-once *delivery* semantics.

See [Producer](./references/producer.md).

---

### 3. Consumer

Consume with `@KafkaListener`, choose manual vs. auto acknowledgment, and keep
listener methods fast and idempotent (consumers may see a record more than once).

See [Consumer](./references/consumer.md).

---

### 4. Retry

Retry **transient** failures only. Prefer non-blocking retry topics
(`@RetryableTopic`) over blocking the consumer thread; use bounded attempts with
exponential backoff.

See [Retry & Backoff](./references/retry.md).

---

### 5. Error Handler

Centralize failure routing in a `DefaultErrorHandler`: classify exceptions as
retryable vs. non-retryable (fatal), and hand exhausted records to the DLQ recoverer.

See [Error Handler](./references/error-handler.md).

---

### 6. DLQ (Dead-Letter Queue)

Send poison messages to a dead-letter topic instead of blocking the partition.
Preserve the original payload + failure metadata so messages can be inspected and
replayed.

See [Dead-Letter Queue](./references/dlq.md).

---

### 7. Integration Test

Prove the end-to-end flow — including retry and DLQ routing — with Testcontainers
(real broker) or `@EmbeddedKafka` (in-memory).

See [Integration Testing](./references/integration-test.md).

---

## Quick Checklist

- [ ] Delivery guarantees chosen (`acks`, idempotence, `auto-offset-reset`)
- [ ] Manual ack + disabled auto-commit for at-least-once consumers
- [ ] Producer sets a partition key and handles the async send result
- [ ] Listener logic is idempotent (safe under redelivery)
- [ ] Retry is bounded, with backoff, and scoped to transient errors only
- [ ] Non-retryable exceptions classified as fatal (no wasted retries)
- [ ] Exhausted/poison messages routed to a DLQ with failure metadata
- [ ] Integration test covers happy path **and** DLQ routing

---

## Reference Files

- [Configuration](./references/configuration.md)
- [Producer](./references/producer.md)
- [Consumer](./references/consumer.md)
- [Retry & Backoff](./references/retry.md)
- [Error Handler](./references/error-handler.md)
- [Dead-Letter Queue](./references/dlq.md)
- [Integration Testing](./references/integration-test.md)