Message Queue Debugging

Message Queue Debugging

Debugs distributed message queue systems including Kafka, RabbitMQ, and SQS — identifies consumer lag, dead letter queue issues, message backlogs, and flow bottlenecks to restore reliable message delivery.

TL;DR Checklist

• Verify queue connectivity with basic connection test for each broker
• Check consumer group status and identify lagging consumers
• Examine dead letter queue (DLQ) for failed message patterns
• Review message throughput metrics (inbound vs outbound rates)
• Validate message visibility timeout settings against processing time
• Analyze partition or queue distribution for load imbalance
• Trace message flow using correlation IDs or message headers
• Check broker logs for errors, warnings, or connection drops

When to Use

Use this skill when:

• Messages are piling up in queues and consumers aren't processing them
• Consumer lag metrics show increasing delays in message processing
• Messages appear in dead letter queues without clear error patterns
• Distributed system messages are disappearing or duplicating unexpectedly
• Queue connection errors or timeouts occur in production logs
• Message ordering guarantees aren't being maintained across consumers

When NOT to Use

Avoid this skill for:

• Simple application logging issues — use agent-logging-debugging instead
• Database query performance problems — use agent-database-admin instead
• Network connectivity outside message broker ports — use cncf-kubernetes-debugging for infrastructure issues
• Application code bugs that don't involve message queue operations
• Authentication/authorization issues — verify IAM roles, ACLs, or user permissions separately

Core Workflow

1. Initial Diagnosis — Gather baseline metrics: queue depths, consumer counts, message rates. Checkpoint: Confirm whether issue is latency (slow processing) or throughput (backlog accumulation).

2. Consumer Health Check — Verify all consumers are connected, active, and not in rebalance state. Checkpoint: Identify any consumers that are offline or stuck in partition assignment.

3. Dead Letter Queue Analysis — Examine DLQ contents to identify failure patterns (serialization errors, business rule violations, infrastructure failures). Checkpoint: Determine if failures are transient (retry) or persistent (data/model issue).

4. Message Flow Tracing — Use correlation IDs or message headers to track specific messages through the system. Checkpoint: Locate exactly where messages are being dropped, duplicated, or delayed.

5. Configuration Review — Validate broker settings: visibility timeouts, retention policies, replication factors, prefetch counts. Checkpoint: Ensure settings match expected processing characteristics.

6. Resolution Implementation — Apply fixes: scale consumers, adjust timeouts, clear DLQ after analysis, restart stuck consumers. Checkpoint: Verify metrics improve within expected time window after changes.

Implementation Patterns

Pattern 1: Kafka Consumer Lag Monitoring

Monitor consumer group lag to detect processing delays.

# Check consumer group lag for all topics
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --list --exclude-internal-topics | \
  xargs -I {} kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group {}

# Detailed lag breakdown by partition
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group my-consumer-group

# ❌ BAD — checking only current offset without lag context
kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic my-topic --from-beginning --max-messages 1

# ✅ GOOD — monitoring lag over time with timestamp
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group my-consumer-group | \
  awk 'NR>3 { lag += $5 } END { print "Total Lag:", lag }'

Pattern 2: Kafka Topic and Partition Analysis

Investigate topic health and partition distribution.

# List all topics with detailed information
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic my-topic

# Check partition leader distribution
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe | awk '/Partition:/{p=$2} /Leader:/{l=$4} END{print "Partitions:", p, "Leaders:", l}'

# ✅ BAD — assuming all partitions are balanced
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe | grep -v "Leader: 0"

# ✅ GOOD — identifying partition imbalance
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe | awk '/Leader:/ {leaders[$4]++} END {for (l in leaders) print l, leaders[l]}'

Pattern 3: RabbitMQ Queue Health Checks

Monitor RabbitMQ queue depths and consumer status.

# List all queues with message counts
rabbitmqctl list_queues name messages messages_ready messages_unacknowledged

# Check specific queue details
rabbitmqctl list_queues name messages messages_ready messages_unacknowledged \
  consumers consumer_utilisation

# ✅ BAD — only checking total messages, ignoring unacknowledged
rabbitmqctl list_queues name messages

# ✅ GOOD — identifying unacknowledged message buildup
rabbitmqctl list_queues name messages_unacknowledged | \
  awk '$2 > 1000 { print "Potential issue:", $1 }'

Pattern 4: RabbitMQ Exchange and Binding Inspection

Debug message routing through exchanges and bindings.

# List all exchanges
rabbitmqctl list_exchanges name type durable internal

# List bindings for a specific exchange
rabbitmqctl list_bindings source_name destination_name destination_type routing_key

# ✅ BAD — not checking if bindings exist for expected routes
rabbitmqctl list_bindings

# ✅ GOOD — verifying binding exists for message routing
rabbitmqctl list_bindings | grep "my-exchange.*my-queue"

Pattern 5: SQS Message Visibility and DLQ Handling

Debug SQS message visibility timeout and dead letter queue issues.

# Check queue attributes including visibility timeout
aws sqs get-queue-attributes --queue-url https://sqs.region.amazonaws.com/123456/MyQueue \
  --attribute-names All

# List dead letter queues and their source queues
aws sqs list-queues --query "QueueUrls[?contains(@, 'DLQ')]"

# ✅ BAD — not checking visibility timeout against processing time
aws sqs send-message --queue-url https://sqs.region.amazonaws.com/123456/MyQueue \
  --message-body "test"

# ✅ GOOD — calculating appropriate visibility timeout
aws sqs get-queue-attributes --queue-url https://sqs.region.amazonaws.com/123456/MyQueue \
  --attribute-names VisibilityTimeout | \
  jq '.Attributes.VisibilityTimeout > (expected_processing_time * 3)'

Pattern 6: Dead Letter Queue Message Analysis

Analyze DLQ contents to identify failure patterns.

# Kafka: Read from DLQ topic
kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic my-topic-dlq --from-beginning --max-messages 100

# RabbitMQ: Consume from dead letter exchange
rabbitmqadmin --vhost=/ get queue=my-queue.DLQ count=10

# SQS: Read messages from DLQ
aws sqs receive-message --queue-url https://sqs.region.amazonaws.com/123456/MyQueueDLQ \
  --max-number-of-messages 10

# ✅ BAD — clearing DLQ without analysis
aws sqs purge-queue --queue-url https://sqs.region.amazonaws.com/123456/MyQueueDLQ

# ✅ GOOD — analyzing DLQ messages for patterns
aws sqs receive-message --queue-url https://sqs.region.amazonaws.com/123456/MyQueueDLQ \
  --max-number-of-messages 10 | \
  jq '.Messages[].Body' | \
  grep -o '"error":"[^"]*"' | \
  sort | uniq -c | sort -rn

Pattern 7: Message Flow Tracing with Correlation IDs

Track specific messages through the distributed system.

# Send message with correlation ID (Kafka)
kafka-console-producer.sh --bootstrap-server localhost:9092 \
  --topic my-topic --property "parse.key=true" \
  --property "key.separator=:" <<< "corr-123:{\"id\":\"corr-123\",\"data\":\"test\"}"

# Read with correlation ID filter
kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic my-topic --from-beginning | \
  jq -r select(.id == "corr-123")

# ✅ BAD — no correlation ID in message headers
kafka-console-producer.sh --bootstrap-server localhost:9092 \
  --topic my-topic <<< '{"data":"test"}'

# ✅ GOOD — end-to-end tracing with correlation ID
# Producer sends: {"correlationId":"req-456", ...}
# Consumer logs: "Processing req-456 - startTime: $(date +%s)"
# Logger stores: correlationId, timestamp, source, destination

Pattern 8: Broker Connection and Health Verification

Verify connectivity and health of message broker infrastructure.

# Kafka: Check broker connectivity
kafka-broker-api-versions.sh --bootstrap-server localhost:9092

# Kafka: Test topic creation and message round-trip
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic test-connection --partitions 1 --replication-factor 1
kafka-console-producer.sh --bootstrap-server localhost:9092 \
  --topic test-connection <<< "test-message"
kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic test-connection --from-beginning --timeout-ms 5000

# ✅ BAD — assuming broker is healthy without verification
# No connectivity check before production deployment

# ✅ GOOD — comprehensive broker health verification
kafka-broker-api-versions.sh --bootstrap-server localhost:9092 \
  | grep -q "kafka" && echo "Broker healthy" || echo "Broker unhealthy"

Additional Debugging Commands

Kafka Advanced Troubleshooting

# Describe consumer group in detail
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group my-group --members

# List consumer group states
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --list | while read group; do
    state=$(kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
      --describe --group "$group" 2>&1 | grep "State:" | head -1)
    echo "$group: $state"
  done

# Check log segments and retention
ls -lh /var/lib/kafka/data/my-topic-0/ | head -20

# Analyze log compaction status
kafka-log-dirs.sh --bootstrap-server localhost:9092 \
  --describe --topic-sets my-topic:0

RabbitMQ Advanced Troubleshooting

# List consumers with details
rabbitmqctl list_consumers queue=my-queue consumer_details

# Check policy application
rabbitmqctl list_policies

# Examine queue length over time (via management API)
curl -s http://localhost:15672/api/queues | jq '.[] | {name, messages, consumers}'

# Force queue synchronization
rabbitmqctl sync_queue my-queue

# Clear a specific queue
rabbitmqadmin purge queue=my-queue

SQS Advanced Troubleshooting

# Check message age in queue
aws sqs get-queue-attributes --queue-url https://sqs.region.amazonaws.com/123456/MyQueue \
  --attribute-names ApproximateAgeOfOldestMessage

# Test message visibility timeout behavior
aws sqs send-message --queue-url https://sqs.region.amazonaws.com/123456/MyQueue \
  --message-body "test" --delay-seconds 0
aws sqs receive-message --queue-url https://sqs.region.amazonaws.com/123456/MyQueue \
  --max-number-of-messages 1 --visibility-timeout 0
# Message should reappear after visibility timeout

# Check redrive policy for DLQ
aws sqs get-queue-attributes --queue-url https://sqs.region.amazonaws.com/123456/MyQueue \
  --attribute-names RedrivePolicy

# List all SQS queues
aws sqs list-queues --query "QueueUrls[]"

General Monitoring Commands

# Monitor queue depth in real-time (Kafka)
watch -n 5 "kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group my-group | grep -v 'GROUP' | awk '{lag+=\$5} END {print lag}'"

# RabbitMQ queue depth monitoring
watch -n 5 "rabbitmqctl list_queues name messages | grep -v 'done'"

# SQS message count monitoring
watch -n 5 "aws sqs get-queue-attributes --queue-url https://sqs.region.amazonaws.com/123456/MyQueue \
  --attribute-names ApproximateNumberOfMessages | jq '.Attributes.ApproximateNumberOfMessages'"

# Check broker disk space
df -h /var/lib/kafka /var/lib/rabbitmq

# Monitor broker logs for errors
tail -f /var/log/kafka/server.log | grep -E "(ERROR|WARN|FATAL)"

Kafka Replication and Leader Election

# Check replication factor and ISR status
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic my-topic | grep -E "ISR|Replicas"

# Trigger preferred leader election
kafka-leader-election.sh --bootstrap-server localhost:9092 \
  --election-type preferred --topic my-topic

# List under-replicated partitions
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe | grep "Under-replicated"

# Check replica lag
kafka-replica-manager.sh --bootstrap-server localhost:9092 \
  --command-config admin.properties --under-replicated-partitions

# List offline partitions
kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe | grep "Offline"

RabbitMQ Cluster and Federation

# Check cluster status
rabbitmqctl cluster_status

# List nodes in cluster
rabbitmqctl list_nodes

# Check federation upstreams
rabbitmqctl list_federation_upstreams

# List federation links
rabbitmqctl list_federation_links

# CheckShovel status
rabbitmqctl list_shovels

# List virtual hosts
rabbitmqctl list_vhosts

# Check memory usage
rabbitmqctl report | grep -A 5 "Memory"

SQS Advanced Message Management

# Set queue policy for DLQ redrive
aws sqs set-queue-attributes --queue-url https://sqs.region.amazonaws.com/123456/MyQueue \
  --attributes '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"arn:aws:sqs:region:123456:MyQueueDLQ\",\"maxReceiveCount\":\"3\"}"}'

# Test message delay
aws sqs send-message --queue-url https://sqs.region.amazonaws.com/123456/MyQueue \
  --message-body "delayed-message" --delay-seconds 60

# Check message retention
aws sqs get-queue-attributes --queue-url https://sqs.region.amazonaws.com/123456/MyQueue \
  --attribute-names MessageRetentionPeriod

# List queue attributes
aws sqs list-queue-attributes --queue-url https://sqs.region.amazonaws.com/123456/MyQueue \
  --attribute-names All

# Get queue ARN
aws sqs get-queue-attributes --queue-url https://sqs.region.amazonaws.com/123456/MyQueue \
  --attribute-names QueueArn

Performance and Throughput Testing

# Kafka: Generate load test
kafka-producer-perf-test.sh --topic test-topic \
  --num-records 10000 --record-size 1024 \
  --throughput 1000 --producer-props bootstrap.servers=localhost:9092

# Kafka: Consumer performance test
kafka-consumer-perf-test.sh --broker-list localhost:9092 \
  --topic test-topic --messages 10000

# RabbitMQ: Message rate test
rabbitmq-perf-test.sh --host localhost --users 10 --queue my-queue \
  --messages 10000 --payload 100

# SQS: Throughput test
for i in {1..1000}; do
  aws sqs send-message --queue-url https://sqs.region.amazonaws.com/123456/MyQueue \
    --message-body "test-$i" &
done
wait

Troubleshooting Consumer Issues

# Kafka: Check consumer offset lag
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group my-group | awk 'NR>3 {total_lag+=$5} END {print total_lag}'

# Kafka: Reset consumer group offset
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group my-group --topic my-topic --reset-offsets --to-latest --execute

# Kafka: Describe consumer group members
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group my-group --members

# Kafka: Check consumer heartbeat status
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group my-group --members | grep "Heartbeat"

# RabbitMQ: Check consumer count per queue
rabbitmqctl list_queues name consumers | awk '$2 > 0 {print}'

# SQS: Check long-polling configuration
aws sqs get-queue-attributes --queue-url https://sqs.region.amazonaws.com/123456/MyQueue \
  --attribute-names ReceiveMessageWaitTimeSeconds

Security and ACL Verification

# Kafka: List ACLs
kafka-acls.sh --authorizer-properties zookeeper=localhost:2181 \
  --list --topic my-topic

# Kafka: Add consumer ACL
kafka-acls.sh --authorizer-properties zookeeper=localhost:2181 \
  --add --allow-principal User:consumer --operation READ \
  --topic my-topic --group my-group

# RabbitMQ: List permissions
rabbitmqctl list_permissions

# RabbitMQ: List user permissions
rabbitmqctl list_user_permissions my-user

# SQS: Get queue policy
aws sqs get-queue-attributes --queue-url https://sqs.region.amazonaws.com/123456/MyQueue \
  --attribute-names Policy

# SQS: Test policy with IAM
aws sqs get-queue-attributes --queue-url https://sqs.region.amazonaws.com/123456/MyQueue \
  --attribute-names policy --output json

Constraints

MUST DO

• Always verify consumer group state before investigating message processing issues
• Correlate DLQ messages with production logs using correlation IDs
• Check visibility timeout settings against actual message processing duration
• Monitor both inbound and outbound message rates to detect bottlenecks
• Verify broker connectivity before assuming application-level issues
• Use the kafka-broker-api-versions.sh command to confirm broker health

MUST NOT DO

• Clear dead letter queues without first analyzing failure patterns
• Increase partition count without understanding message key distribution
• Ignore consumer rebalance events — they indicate instability
• Set visibility timeout shorter than expected maximum processing time
• Assume message loss without tracing correlation IDs through the system
• Disable DLQ functionality to "fix" message accumulation

Output Template

When debugging a message queue issue, the output must contain:

1. Issue Classification — Is this latency (slow processing) or throughput (backlog)? Include current queue depth and consumer count.

2. Root Cause Analysis — Based on diagnostic results, identify the specific failure mode (e.g., "consumer stuck in rebalance", "DLQ filling due to deserialization errors").

3. Immediate Actions Taken — List all commands executed and their output, including any fixes applied.

4. Verification Steps — Commands to confirm the issue is resolved and metrics have improved.

5. Prevention Measures — Configuration changes or monitoring additions to prevent recurrence.

Related Skills

References

• Kafka Documentation: https://kafka.apache.org/documentation/
• RabbitMQ Documentation: https://www.rabbitmq.com/documentation.html
• AWS SQS Documentation: https://docs.aws.amazon.com/sqs/
• Kafka CLI Tools: kafka-consumer-groups.sh, kafka-topics.sh, kafka-broker-api-versions.sh
• RabbitMQ CLI Tools: rabbitmqctl, rabbitmqadmin
• AWS CLI: aws sqs commands for SQS management

📖 skill: message-queue-debugging