incidentfox Kafka Streaming
Authentication
IMPORTANT: Credentials are injected automatically by a proxy layer. Do NOT check for KAFKA_SASL_PASSWORD in environment variables - it won't be visible to you. Just run the scripts directly; authentication is handled transparently.
Configuration environment variables you CAN check (non-secret):
KAFKA_BOOTSTRAP_SERVERS- Kafka broker addressesKAFKA_SECURITY_PROTOCOL- Security protocol (PLAINTEXT, SSL, SASL_SSL, SASL_PLAINTEXT)
MANDATORY: Broker-First Investigation
Start with broker info, then check topics and consumer groups.
BROKER INFO → LIST TOPICS → DESCRIBE TOPIC → CHECK CONSUMER LAG
Available Scripts
All scripts are in .claude/skills/streaming-kafka/scripts/
get_broker_info.py - ALWAYS START HERE
python .claude/skills/streaming-kafka/scripts/get_broker_info.py
list_topics.py - List Topics
python .claude/skills/streaming-kafka/scripts/list_topics.py [--include-internal]
describe_topic.py - Topic Details with Offsets
python .claude/skills/streaming-kafka/scripts/describe_topic.py --topic TOPIC_NAME
list_consumer_groups.py - List Consumer Groups
python .claude/skills/streaming-kafka/scripts/list_consumer_groups.py
describe_consumer_group.py - Consumer Group Details
python .claude/skills/streaming-kafka/scripts/describe_consumer_group.py --group GROUP_ID
get_consumer_lag.py - Consumer Lag with Health Assessment
python .claude/skills/streaming-kafka/scripts/get_consumer_lag.py --group GROUP_ID [--topic TOPIC]
Consumer Lag Health Levels
| Total Lag | Health |
|---|---|
| 0 | healthy |
| < 1,000 | minor_lag |
| < 100,000 | lagging |
| >= 100,000 | severely_lagging |
Investigation Workflow
Consumer Lag Investigation
1. get_broker_info.py (verify cluster health)
2. list_consumer_groups.py (find the group)
3. get_consumer_lag.py --group <group-id> (check lag)
4. describe_topic.py --topic <topic> (check partition details)
Topic Issue Investigation
1. list_topics.py
2. describe_topic.py --topic <topic> (partitions, configs, offsets)
3. Check under-replicated partitions in output