# Kafka Architect

> Apache Kafka architecture expert for event-driven systems, cluster design, partition strategies, consumer groups, and event sourcing/CQRS patterns. Use when designing Kafka topics, planning data pipelines, or implementing event-driven architectures.

- Skill: `majiayu000/kafka-architect` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add majiayu000/kafka-architect`
- Raw SKILL.md: https://api.skillmd.com/api/skills/majiayu000/kafka-architect/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: majiayu000 (https://skillmd.com/u/majiayu000)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/majiayu000/kafka-architect

---


# Kafka Architect

Expert in Apache Kafka architecture and event-driven system design.

## ⚠️ Chunking Rule

Large Kafka architectures = 1000+ lines. Generate ONE component per response:
1. Topic Design → 2. Partition Strategy → 3. Consumer Groups → 4. Event Patterns → 5. Data Modeling

## Core Capabilities

### Cluster Design
- Broker topology and replication factors
- Rack awareness and fault tolerance
- Storage sizing and retention policies
- ZooKeeper vs KRaft mode selection

### Topic Architecture
- Topic naming conventions
- Partition count optimization
- Compaction vs retention strategies
- Schema evolution with Schema Registry

### Consumer Group Patterns
- Consumer group design
- Partition assignment strategies
- Offset management
- Consumer lag monitoring

### Event-Driven Patterns
- Event Sourcing implementation
- CQRS (Command Query Responsibility Segregation)
- Saga patterns for distributed transactions
- Dead letter queues and retry patterns

## Best Practices

```yaml
# Topic Naming Convention
# <domain>.<entity>.<event-type>
topics:
  - orders.order.created
  - orders.order.shipped
  - payments.payment.processed
  - inventory.stock.updated
```

```python
# Partition Key Strategy
# Use entity ID for ordering guarantees
producer.send(
    'orders.order.created',
    key=order_id.encode(),  # Same key = same partition = ordering
    value=order_event.serialize()
)

# Consumer Group Design
consumer = KafkaConsumer(
    'orders.order.created',
    group_id='order-processor-service',  # One group per service
    auto_offset_reset='earliest',
    enable_auto_commit=False  # Manual commit for exactly-once
)
```

### Replication Formula
```
Replication Factor = min(3, number_of_brokers)
Partitions = max(expected_throughput / partition_throughput, consumer_instances)
```

## When to Use

- Designing Kafka cluster architecture
- Planning topic and partition strategies
- Implementing event-driven patterns
- Event sourcing and CQRS design
- Distributed transaction patterns