# Kafka Streaming

> Kafka topic and consumer group management. Use when investigating Kafka topics, consumer lag, broker health, or consumer group status.

- Skill: `incidentfox/kafka-streaming` (Agent Skill, multi-file: 8 files)
- Install (CLI): `npx skillmds@latest add incidentfox/kafka-streaming`
- Raw SKILL.md: https://api.skillmd.com/api/skills/incidentfox/kafka-streaming/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: incidentfox (https://skillmd.com/u/incidentfox)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/incidentfox/kafka-streaming

---


# Kafka Streaming

## Authentication

**IMPORTANT**: Credentials are injected automatically by a proxy layer. Do NOT check for `KAFKA_SASL_PASSWORD` in environment variables - it won't be visible to you. Just run the scripts directly; authentication is handled transparently.

Configuration environment variables you CAN check (non-secret):
- `KAFKA_BOOTSTRAP_SERVERS` - Kafka broker addresses
- `KAFKA_SECURITY_PROTOCOL` - Security protocol (PLAINTEXT, SSL, SASL_SSL, SASL_PLAINTEXT)

---

## MANDATORY: Broker-First Investigation

**Start with broker info, then check topics and consumer groups.**

```
BROKER INFO → LIST TOPICS → DESCRIBE TOPIC → CHECK CONSUMER LAG
```

## Available Scripts

All scripts are in `.claude/skills/streaming-kafka/scripts/`

### get_broker_info.py - ALWAYS START HERE
```bash
python .claude/skills/streaming-kafka/scripts/get_broker_info.py
```

### list_topics.py - List Topics
```bash
python .claude/skills/streaming-kafka/scripts/list_topics.py [--include-internal]
```

### describe_topic.py - Topic Details with Offsets
```bash
python .claude/skills/streaming-kafka/scripts/describe_topic.py --topic TOPIC_NAME
```

### list_consumer_groups.py - List Consumer Groups
```bash
python .claude/skills/streaming-kafka/scripts/list_consumer_groups.py
```

### describe_consumer_group.py - Consumer Group Details
```bash
python .claude/skills/streaming-kafka/scripts/describe_consumer_group.py --group GROUP_ID
```

### get_consumer_lag.py - Consumer Lag with Health Assessment
```bash
python .claude/skills/streaming-kafka/scripts/get_consumer_lag.py --group GROUP_ID [--topic TOPIC]
```

---

## Consumer Lag Health Levels
| Total Lag | Health |
|-----------|--------|
| 0 | healthy |
| < 1,000 | minor_lag |
| < 100,000 | lagging |
| >= 100,000 | severely_lagging |

---

## Investigation Workflow

### Consumer Lag Investigation
```
1. get_broker_info.py (verify cluster health)
2. list_consumer_groups.py (find the group)
3. get_consumer_lag.py --group <group-id> (check lag)
4. describe_topic.py --topic <topic> (check partition details)
```

### Topic Issue Investigation
```
1. list_topics.py
2. describe_topic.py --topic <topic> (partitions, configs, offsets)
3. Check under-replicated partitions in output
```