Apache Kafka Knowledge Patch

Apache Kafka Knowledge Patch

Use this skill when upgrading, configuring, extending, or operating Apache Kafka and its clients, Kafka Streams, Kafka Connect, MirrorMaker, KRaft, share groups, or tiered storage. Inspect the project's actual broker, client, metadata, and Java versions before applying version-dependent advice.

Reference index

Upgrade safety first

Establish the cluster mode and floor

• Kafka brokers are KRaft-only. Migrate any ZooKeeper cluster before attempting a broker upgrade.
• Do not roll an old KRaft cluster directly when its software or metadata version is below 3.3. Move it to 3.9.x first.
• Before a 4.0 broker roll, require Kafka clients, Streams, and Connect at 2.1 or later; exactly-once Streams workloads require brokers 2.5 or later.
• Verify non-Apache client compatibility independently.
• Brokers, Connect, and Kafka tools require Java 17; clients and Streams require Java 11. Java 25 is supported by Kafka 4.2.

Prefer fixed maintenance releases

• Avoid Kafka Streams 4.1.0 because range scans, windows, and joins can leak memory; use 4.1.1 or later.
• Use at least 4.2.1 for share-group deadlock fixes, transactional rolling-upgrade safety, and offline classic-to-Streams group migration.
• Avoid Kafka Streams 4.3.0 because cascading task closes can leak RocksDB column-family resources; use 4.3.1 or later.

Finalize deliberately

After rolling every broker and validating cluster health, finalize the intended release:

bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --release-version 4.3

• Do not assume rolling new binaries changes every protocol. Feature finalization activates metadata and protocol behavior.
• Metadata downgrade is impossible across feature levels that contain metadata changes, including 4.0 and 4.3.
• The 4.2 metadata level itself contains no metadata changes, but downgrade is safe only if no crossed intermediate level contains them.
• Inspect finalized, minimum-supported, and maximum-supported feature metrics before planning a downgrade.

Breaking configuration changes

Broker and common configuration

• Remove log.message.format.version, message.format.version, offsets.commit.required.acks, and log.message.timestamp.difference.max.ms.
• Replace the timestamp-difference limit with log.message.timestamp.before.max.ms and log.message.timestamp.after.max.ms. The latter defaults to one hour and can reject future-dated CreateTime records.
• Replace metrics.jmx.blacklist/whitelist with metrics.jmx.exclude/include.
• Replace delegation.token.master.key with delegation.token.secret.key.
• Replace remote.log.manager.thread.pool.size with remote.log.manager.follower.thread.pool.size.
• Treat LIST settings strictly: null, duplicates, and empty lists may now be rejected or warned about according to each setting.

Logging and metrics

• Kafka uses Log4j2. Convert old configuration with log4j-transform-cli and replace KafkaLog4jAppender with the Log4j2 Kafka appender.
• Update custom cleaner logging to the org.apache.kafka.storage.internals.log logger names.
• Migrate nanosecond totals to bufferpool-wait-time-ns-total, io-wait-time-ns-total, and io-time-ns-total.
• Prefer AppInfo metrics tagged with client-id; the untagged forms are being removed.
• Move monitoring to unchanged consumer topic names rather than dot-to-underscore variants.

Client migration essentials

Producer and transactions

• Account for the producer linger.ms default of 5.
• Idempotence no longer silently falls back when max.in.flight.requests.per.connection exceeds 5; fix the configuration.
• Never call KafkaProducer.flush() from a producer callback.
• Treat both TimeoutException and TransactionAbortableException as transaction-abort conditions; do not retry the same transaction and risk duplicates.
• Replace the removed sendOffsetsToTransaction(..., String consumerGroupId) overload with the group-metadata form.

Consumer and Admin

• Replace Consumer.poll(long) with poll(Duration) and account for its strict timeout during assignment.
• Use set-based committed(...) overloads and MockConsumer.setPollException().
• Migrate Admin.alterConfigs() to incrementalAlterConfigs().
• Use GroupState, type-neutral group listing, UUID-aware topic APIs, and the current result accessor names.
• Prefer Consumer.close(CloseOptions); close(Duration) is deprecated.

Group protocols

• Use kafka-groups.sh for cross-type group discovery.
• Treat share groups as record-level cooperative work queues, not ordered partition streams.
• For clusters with fewer than three brokers, set share-state topic replication and minimum ISR before the first share group creates __share_group_state.
• Enable protocols through group.version, streams.version, and share.version feature levels; group.coordinator.rebalance.protocols is deprecated.
• Plan migration away from the classic consumer rebalance protocol.

Kafka Streams essentials

Removed and renamed APIs

• Migrate off the old Processor API, transformers, KStream.through(), KStream.branch(), old window builders, and the old uncaught-exception handler.
• Follow the dedicated transformValues() to processValues() migration; it is not a mechanical rename and requires the 4.0.1 fix.
• Use deserialization.exception.handler and production.exception.handler.
• Use top-level org.apache.kafka.streams.CloseOptions.
• Migrate ProductionExceptionHandlerResponse to ProductionExceptionHandler.Response and the handler methods to handleError and handleSerializationError.

Rebalance protocol

• The broker-driven Streams protocol is production-ready in 4.2, but offline migration from classic groups needs 4.2.1.
• Do not reuse an application.id that identifies a classic Streams app or another group when testing the early protocol.
• Administer Streams groups with kafka-streams-groups.sh; protocol-owned session, heartbeat, and standby settings are group configurations.
• Significant topology changes may require a new Streams group.

State and downgrade safety

• Before downgrading from 4.0.x after RocksDB format migration, delete local state and restore it from changelogs.
• Kafka Streams 4.3 stores changelog offsets in state stores. RocksDB uses an offsets column family that older releases cannot open.
• Before downgrading from 4.3.x, stop each instance, delete its local state.dir, and restore from changelogs.
• Header-aware stores are opt-in. A rolling upgrade works, but migrated local data likewise requires clearing before downgrade.

Exception handling and DLQs

• A ProductionExceptionHandler is called for retriable producer timeouts; explicitly choose retry, continue, or fail.
• Configure errors.dead.letter.queue.topic.name when using DefaultProductionExceptionHandler; custom handlers must build DLQ records themselves.
• Enable global-store processing exception handling with processing.exception.handler.global.enabled=true; global-thread DLQ handling is not available.

Connect and MirrorMaker essentials

• Connect requires Java 17 and Jakarta EE 10-compatible integrations.
• Replace GET /connectors/{connector}/tasks-config with GET /connectors/{connector}/tasks.
• Use include/exclude instead of removed whitelist/blacklist settings.
• Do not override the removed task assignment hooks or single-argument SourceTask.commitRecord hook.
• MirrorMaker 1 is removed; use MirrorMaker 2 and brokers that support incremental alter configs.
• Use MirrorMaker exclude settings and plan migration to the new metric-name formats.
• Use side-by-side Connect plugin versions for controlled upgrades and rollbacks.

CLI migration essentials

• Pass --bootstrap-server endpoints as one comma-separated value.
• Replace whitelist-style options with --include or --topics-include.
• Migrate generic --property, --config, and command-property options to each tool's typed replacement.
• Custom console readers and dump-log decoders must implement the APIs under org.apache.kafka.tools.api.
• Use the current formatter packages under org.apache.kafka.tools.consumer.
• Treat kafka-topics --delete-config as deprecated; kafka-configs.sh --alter --delete-config is idempotent for missing keys.

Operational checks

Before declaring an upgrade complete:

1. Confirm broker software, metadata, and production feature levels on every node.
2. Verify Java runtimes and KRaft-only configuration.
3. Check group coordinators, transaction workloads, client rebootstrap, and rack-aware assignment.
4. Verify remote-log metadata ISR, follower pools, and tiered-storage behavior.
5. Compare metric names, domains, tags, and ratio semantics with dashboards and alerts.
6. Exercise Connect plugins, MirrorMaker replication policy, and custom security callbacks.
7. Restart Streams workloads from both clean and existing local state, then validate restore and downgrade procedures.

Consult the topic references for exact APIs, defaults, limitations, and version-specific attribution before editing production configuration or code.