# Kafka

> Build, review, debug, configure, migrate, teach, or plan Apache Kafka work from TypeScript with current docs. Prefer @platformatic/kafka as the TS SDK (Producer, Consumer, Admin, bootstrapBrokers, consume streams, ProduceAcks, transactions). Use for topics, partitions, consumer groups, offsets, acks, idempotence, EOS, compaction, retention, SASL/TLS, Schema Registry concepts, KRaft, Kafka 4.x, kafkajs migration, DLQ/outbox patterns, and broker ops that affect app produce/consume.

- Skill: `dobroslavradosavljevic/kafka` (Agent Skill, multi-file: 8 files)
- Install (CLI): `npx skillmds@latest add dobroslavradosavljevic/kafka`
- Raw SKILL.md: https://api.skillmd.com/api/skills/dobroslavradosavljevic/kafka/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: dobroslavradosavljevic (https://skillmd.com/u/dobroslavradosavljevic)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/dobroslavradosavljevic/kafka

---


# Apache Kafka (TypeScript)

Use this skill for Kafka produce/consume/admin work driven from TypeScript/Node: prefer **`@platformatic/kafka`**, plus broker semantics, topic design, delivery guarantees, and ops that change app code.

## Workflow

1. Inspect the local surface:
   - Package: **`@platformatic/kafka`** (snapshot **2.8.0**). Engines: Node **`>=22.22` or `>=24.6`**. Kafka brokers **3.5–4.2** (KRaft-only on Kafka **4.0+**).
   - Clients: `Producer` / `Consumer` / `Admin` with `bootstrapBrokers` (not KafkaJS `brokers`).
   - Topics: partitions, keys, RF / `min.insync.replicas`, cleanup policy.
   - Semantics: at-least-once vs EOS; auto vs manual commit; `ProduceAcks`.
2. For day-to-day how-to, follow [usage-guide.md](references/usage-guide.md) first.
3. Refresh docs when versions drift. Start from [source-map.md](references/source-map.md).
4. Route deeper detail:
   - Concepts (partitions, groups, retention, EOS): [fundamentals.md](references/fundamentals.md).
   - `@platformatic/kafka` API, serializers, streams, txns, SASL: [platformatic-kafka.md](references/platformatic-kafka.md).
   - Design, DLQ, outbox, monitoring: [patterns-ops.md](references/patterns-ops.md).
   - Client landscape & brokers: [clients-ecosystem.md](references/clients-ecosystem.md).
5. Prefer **`@platformatic/kafka`** over `kafkajs` (unmaintained) and over native `node-rdkafka` unless librdkafka is required. Do not invent KafkaJS `connect`/`subscribe`/`run` APIs on Platformatic.
6. Verify with focused produce/consume smokes, Admin topic create when needed, lag awareness, and `close()` on shutdown.

## Core Judgment

- Kafka is an **append-only partitioned log**. Ordering is **per partition** only; parallelism ceiling for a group = partition count.
- Default app semantic: **at-least-once** + **idempotent handlers**. Use transactions/`READ_COMMITTED` only when you need Kafka↔Kafka EOS.
- Durability triad: RF=3, `min.insync.replicas=2`, producer **`ProduceAcks.ALL`** (+ idempotent producer when available).
- Keys: stable entity ids for order/partitioning. Compaction requires keys + tombstones (`null` value).
- Prefer **outbox/CDC** over dual-write (DB + Kafka in one request).
- Poison pills: retry/DLQ then commit — never block a partition forever.
- Platformatic: long-lived clients; lazy connect; **`close()`**; consume via **`consume()` → stream** (`for await` / `data` / concurrent `forEach`); offsets are **`bigint`**; headers on consume are a **`Map`**.
- Pass **`serializers` / `deserializers`** (e.g. `stringSerializers`) — default expects `Buffer`.
- Kafka **4.0+** is **KRaft-only** (no ZooKeeper). Local/dev: fix `advertised.listeners`.
- Prefer **`bun` / `bunx`** in command examples; runtime must still satisfy Node engines for `@platformatic/kafka`.

## Verification

Prefer repository-owned commands. For meaningful Kafka work, cover the relevant subset:

- `bun pm ls @platformatic/kafka` and Node ≥22.22 (or ≥24.6).
- Smoke: Admin `createTopics` (or existing topic), `Producer.send`, `Consumer.consume` + process + `close()`.
- Durability: `ProduceAcks.ALL`; idempotence/`transactionalId` when claiming EOS.
- Consumer: correct `groupId`, mode (`EARLIEST`/`LATEST`/`COMMITTED`), commit-after-process when `autocommit: false`.
- Security: TLS + SASL (or mTLS); no PLAINTEXT on public networks.
- Ops: consumer lag, under-replicated partitions, hot-key skew, rebalance storms (`max.poll` / processing time).

Report which checks ran, which did not, and broker/version assumptions that remain.