Kubeblocks Addon Kafka

Deploy Kafka on KubeBlocks

Legacy compatibility shim. Primary entry: kubeblocks-engine-kafka. Keep the preserved workflow below for detailed reference, but do not recommend this skill as the main path for cold-start agents.

Overview

Deploy Apache Kafka clusters on Kubernetes using KubeBlocks. Supports combined mode (broker + controller in one process) and separated mode (dedicated broker and controller nodes) with optional monitoring via exporter.

Official docs: https://kubeblocks.io/docs/preview/user_docs/kubeblocks-for-kafka/cluster-management/create-a-kafka-cluster Full doc index: https://kubeblocks.io/llms-full.txt

Prerequisites

• A running Kubernetes cluster with KubeBlocks installed (see install-kubeblocks)
• The Kafka addon must be enabled:

# Check if kafka addon is installed
helm list -n kb-system | grep kafka

# Install if missing
helm install kb-addon-kafka kubeblocks/kafka --namespace kb-system --version 1.0.0

Available Topologies

Kafka uses KRaft mode (no ZooKeeper dependency). Controllers manage cluster metadata via the Raft protocol.

Supported Versions

Workflow

- [ ] Step 1: Ensure addon is installed
- [ ] Step 2: Create namespace
- [ ] Step 3: Create cluster (choose topology)
- [ ] Step 4: Wait for cluster to be ready
- [ ] Step 5: Produce and consume messages

Step 1: Ensure Addon Is Installed

helm list -n kb-system | grep kafka

If not found:

helm install kb-addon-kafka kubeblocks/kafka --namespace kb-system --version 1.0.0

Step 2: Create Namespace

kubectl create namespace demo --dry-run=client -o yaml | kubectl apply -f -

Step 3: Create Cluster

Combined Mode (Dev/Test)

Broker and controller run in the same process. Simpler but not recommended for production:

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: kafka-cluster
  namespace: demo
spec:
  clusterDef: kafka
  topology: combined
  terminationPolicy: Delete
  componentSpecs:
    - name: kafka-combine
      serviceVersion: "3.3.2"
      replicas: 3
      resources:
        limits: {cpu: "0.5", memory: "0.5Gi"}
        requests: {cpu: "0.5", memory: "0.5Gi"}
      env:
        - name: KB_KAFKA_BROKER_HEAP
          value: "-Xmx256m -Xms256m"
        - name: KB_KAFKA_CONTROLLER_HEAP
          value: "-Xmx256m -Xms256m"
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes: [ReadWriteOnce]
            resources: {requests: {storage: 20Gi}}
        - name: metadata
          spec:
            accessModes: [ReadWriteOnce]
            resources: {requests: {storage: 5Gi}}

Separated Mode (Production)

Dedicated broker and controller nodes for better isolation and scalability:

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: kafka-separated
  namespace: demo
spec:
  clusterDef: kafka
  topology: separated
  terminationPolicy: Delete
  componentSpecs:
    - name: kafka-broker
      serviceVersion: "3.3.2"
      replicas: 3
      resources:
        limits: {cpu: "0.5", memory: "0.5Gi"}
        requests: {cpu: "0.5", memory: "0.5Gi"}
      env:
        - name: KB_KAFKA_BROKER_HEAP
          value: "-Xmx256m -Xms256m"
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes: [ReadWriteOnce]
            resources: {requests: {storage: 20Gi}}
        - name: metadata
          spec:
            accessModes: [ReadWriteOnce]
            resources: {requests: {storage: 5Gi}}
    - name: kafka-controller
      serviceVersion: "3.3.2"
      replicas: 3
      resources:
        limits: {cpu: "0.5", memory: "0.5Gi"}
        requests: {cpu: "0.5", memory: "0.5Gi"}
      env:
        - name: KB_KAFKA_CONTROLLER_HEAP
          value: "-Xmx256m -Xms256m"
      volumeClaimTemplates:
        - name: metadata
          spec:
            accessModes: [ReadWriteOnce]
            resources: {requests: {storage: 5Gi}}

Key points:

• Brokers need both data and metadata volumes
• Controllers only need a metadata volume (they don't store message data)
• 3 controller replicas for Raft consensus

Separated Mode with Monitoring

Add an exporter component for Prometheus-compatible metrics:

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: kafka-monitored
  namespace: demo
spec:
  clusterDef: kafka
  topology: separated_monitor
  terminationPolicy: Delete
  componentSpecs:
    - name: kafka-broker
      serviceVersion: "3.3.2"
      replicas: 3
      resources:
        limits: {cpu: "0.5", memory: "0.5Gi"}
        requests: {cpu: "0.5", memory: "0.5Gi"}
      env:
        - name: KB_KAFKA_BROKER_HEAP
          value: "-Xmx256m -Xms256m"
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes: [ReadWriteOnce]
            resources: {requests: {storage: 20Gi}}
        - name: metadata
          spec:
            accessModes: [ReadWriteOnce]
            resources: {requests: {storage: 5Gi}}
    - name: kafka-controller
      serviceVersion: "3.3.2"
      replicas: 3
      resources:
        limits: {cpu: "0.5", memory: "0.5Gi"}
        requests: {cpu: "0.5", memory: "0.5Gi"}
      env:
        - name: KB_KAFKA_CONTROLLER_HEAP
          value: "-Xmx256m -Xms256m"
      volumeClaimTemplates:
        - name: metadata
          spec:
            accessModes: [ReadWriteOnce]
            resources: {requests: {storage: 5Gi}}
    - name: kafka-exporter
      replicas: 1
      resources:
        limits: {cpu: "0.5", memory: "0.5Gi"}
        requests: {cpu: "0.5", memory: "0.5Gi"}

Environment Variables

Kafka configuration can be tuned via environment variables on the component spec:

Step 4: Wait for Cluster Ready

kubectl -n demo get cluster <cluster-name> -w

Wait until STATUS shows Running. Kafka clusters typically take 2-4 minutes.

Check pods:

kubectl -n demo get pods -l app.kubernetes.io/instance=<cluster-name>

Step 5: Produce and Consume Messages

Create a Topic

kubectl -n demo exec -it kafka-cluster-kafka-combine-0 -- \
  kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test-topic --partitions 3 --replication-factor 1

Produce Messages

kubectl -n demo exec -it kafka-cluster-kafka-combine-0 -- \
  kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test-topic

Consume Messages

kubectl -n demo exec -it kafka-cluster-kafka-combine-0 -- \
  kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning

For Separated Mode

Replace the pod name with the broker pod:

kubectl -n demo exec -it kafka-separated-kafka-broker-0 -- \
  kafka-topics.sh --bootstrap-server localhost:9092 --list

Troubleshooting

Cluster stuck in Creating:

kubectl -n demo describe cluster <cluster-name>
kubectl -n demo get events --sort-by='.lastTimestamp'

Broker not starting:

kubectl -n demo logs <broker-pod>

Controller quorum issues:

• Ensure exactly 3 controller replicas for Raft consensus
• Check controller logs for election errors

Out of memory:

• Adjust heap settings via KB_KAFKA_BROKER_HEAP / KB_KAFKA_CONTROLLER_HEAP environment variables
• Ensure container memory limits are larger than heap size

Day-2 Operations

Safety Patterns

Follow safety-patterns.md for dry-run before apply, status confirmation after watch, and pre-deletion checklist.

Next Steps

• For full YAML examples of all topologies (combined_monitor, separated_monitor with production sizing), environment variable reference, and component/volume matrix, see reference.md