Message Queue Troubleshooting

Message Queue Troubleshooting

Diagnoses and resolves issues across Kafka, RabbitMQ, SQS, and NATS message queues with real diagnostic commands, dead letter queue analysis, visibility timeout fixes, and backlog handling strategies.

TL;DR Checklist

• Check cluster health before investigating individual queues
• Verify consumer group status and lag metrics
• Inspect dead letter queues for error patterns
• Validate visibility timeout settings against processing duration
• Analyze message backlog trends before scaling
• Confirm network connectivity to queue brokers
• Review broker logs for warnings and errors
• Test with sample messages after fixes

When to Use

Use this skill when:

• Kafka consumers are not processing messages and you need to identify if it's a consumer group issue or broker problem
• RabbitMQ queues show message accumulation and you need to diagnose producer/consumer imbalances
• SQS messages become invisible but aren't being processed, indicating visibility timeout issues
• Message backlogs are growing and you need to determine root cause (slow consumers, high producers, or failures)
• Dead letter queues are filling and you need to understand why messages are being rejected
• NATS streaming clusters show message persistence issues or subscription problems

When NOT to Use

Avoid this skill for:

• Application-level message handling bugs — use application debugging skills instead
• Message format validation issues — use data validation skills instead
• Simple message loss without queue infrastructure involvement
• Cases where you don't have access to queue broker admin commands
• Real-time trading scenarios where queue delay is expected — use monitoring skills instead

Core Workflow

1. Assess Cluster Health — Check broker connectivity and overall cluster status. Checkpoint: All brokers responding and cluster state is healthy before proceeding.

2. Identify Affected Queues — List all queues and identify which are showing issues (backlog, no consumers, errors). Checkpoint: You have a clear list of problematic queues with their metrics.

3. Check Consumer Status — Verify consumers are connected, active, and not experiencing rebalances. Checkpoint: Consumer groups or queue consumers show expected state (connected, no errors).

4. Inspect Dead Letter Queues — Examine DLQ for failed messages and identify error patterns. Checkpoint: You understand the failure pattern causing messages to reach DLQ.

5. Analyze Message Backlog — Determine backlog growth rate and correlate with processing capacity. Checkpoint: You can distinguish between healthy accumulation and problematic backlog.

6. Apply Resolution Strategy — Implement fixes based on root cause (visibility timeout, scaling, DLQ processing). Checkpoint: Changes applied and verified with test message flow.

Implementation Patterns

Pattern 1: Kafka Cluster Diagnosis

Problem: Kafka consumers not processing messages, potential consumer group or partition issues.

# 1. Check cluster connectivity
kafka-broker-api-versions --bootstrap-server localhost:9092

# 2. List all topics
kafka-topics --bootstrap-server localhost:9092 --list

# 3. Describe topic to see partitions and replicas
kafka-topics --bootstrap-server localhost:9092 --describe --topic your-topic-name

# 4. Check consumer groups status
kafka-consumer-groups --bootstrap-server localhost:9092 --list

# 5. Inspect consumer group lag (critical metric)
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group your-consumer-group

# 6. View recent messages on topic (debugging only)
kafka-console-consumer --bootstrap-server localhost:9092 --topic your-topic-name --from-beginning --max-messages 100

# 7. Check broker logs for errors
tail -f /var/log/kafka/server.log | grep -i error

# 8. Verify ZooKeeper connectivity (for older Kafka versions)
kafka-zookeeper-shutil --bootstrap-server localhost:2181 --describe-node

BAD: Running consumer commands without checking cluster connectivity first

# ❌ BAD — will fail silently if broker is down
kafka-consumer-groups --bootstrap-server localhost:9092 --list

GOOD: Always verify cluster connectivity first

# ✅ GOOD — verify broker is responding
kafka-broker-api-versions --bootstrap-server localhost:9092
# If this works, proceed with consumer commands
kafka-consumer-groups --bootstrap-server localhost:9092 --list

Pattern 2: RabbitMQ Queue Debugging

Problem: RabbitMQ queues accumulating messages, consumers not processing, dead letter issues.

# 1. Check RabbitMQ status
rabbitmqctl status

# 2. List all queues with message counts
rabbitmqctl list_queues name messages messages_ready messages_unacknowledged consumers

# 3. Describe specific queue
rabbitmqctl list_queues name messages messages_ready messages_unacknowledged consumers policy durable

# 4. List exchanges
rabbitmqctl list_exchanges name type durable internal

# 5. List bindings for a queue
rabbitmqctl list_bindings source_name destination_name destination_type routing_key

# 6. List consumers per queue
rabbitmqctl list_consumers queue_name consumer_details

# 7. Clear queue (DANGEROUS — use only in development)
rabbitmqctl purge_queue your-queue-name

# 8. List dead letter queues
rabbitmqctl list_queues name messages | grep dlq

# 9. Check broker logs
tail -f /var/log/rabbitmq/rabbit@*.log

# 10. Export queue definitions for analysis
rabbitmqctl export_definitions /tmp/definitions.json

BAD: Ignoring unacknowledged messages when diagnosing backlogs

# ❌ BAD — only shows ready messages, misses unacked that block processing
rabbitmqctl list_queues name messages

GOOD: Include all message states for complete picture

# ✅ GOOD — shows ready, unacked, and total for full picture
rabbitmqctl list_queues name messages messages_ready messages_unacknowledged
# If unacked > 0, consumers are not acking — check consumer code

Pattern 3: SQS Visibility Timeout Fix

Problem: SQS messages become invisible but aren't processed, indicating visibility timeout issues.

# 1. List all queues
aws sqs list-queues

# 2. Get queue attributes (critical for visibility timeout)
aws sqs get-queue-attributes --queue-url https://sqs.region.amazonaws.com/account/queue-name --attribute-names All

# 3. Check specific attributes
aws sqs get-queue-attributes --queue-url https://sqs.region.amazonaws.com/account/queue-name --attribute-names VisibilityTimeout,MaximumMessageSize,MessageRetentionPeriod

# 4. View invisible messages (requires visibility timeout to expire first)
aws sqs receive-message --queue-url https://sqs.region.amazonaws.com/account/queue-name --max-number-of-messages 10 --attribute-names All

# 5. Check for dead letter queue configuration
aws sqs get-queue-attributes --queue-url https://sqs.region.amazonaws.com/account/queue-name --attribute-names RedrivePolicy

# 6. View DLQ messages
aws sqs receive-message --queue-url https://sqs.region.amazonaws.com/account/queue-name-dlq --max-number-of-messages 10

# 7. Calculate visibility timeout issue
# Formula: processing_time > visibility_timeout → message becomes visible again
# Example: 5-minute processing but 30-second timeout = message loops

# 8. Update visibility timeout
aws sqs set-queue-attributes --queue-url https://sqs.region.amazonaws.com/account/queue-name --attributes '{"VisibilityTimeout": "300"}'

# 9. Check SQS metrics in CloudWatch
aws cloudwatch get-metric-statistics --namespace AWS/SQS --metric-name ApproximateNumberOfMessagesVisible --dimensions Name=QueueName,Value=your-queue-name --start-time 2024-01-01T00:00:00Z --end-time 2024-01-01T01:00:00Z --period 300 --statistics Average

# 10. Send test message
aws sqs send-message --queue-url https://sqs.region.amazonaws.com/account/queue-name --message-body "test message"

BAD: Assuming invisible messages are being processed

# ❌ BAD — invisible messages may be stuck, not processed
aws sqs get-queue-attributes --queue-url URL --attribute-names ApproximateNumberOfMessages
# This only shows VISIBLE messages, not invisible stuck ones

GOOD: Check both visible and invisible counts

# ✅ GOOD — check both metrics
aws sqs get-queue-attributes --queue-url URL --attribute-names ApproximateNumberOfMessages,ApproximateNumberOfMessagesNotVisible
# If NotVisible > 0 and Visible = 0, messages are stuck in processing

Pattern 4: NATS Streaming Diagnostic

Problem: NATS streaming subscriptions not receiving messages, persistence issues, channel backlogs.

# 1. Check NATS server status (for NATS 2.x)
nats server ps

# 2. List streams (for NATS JetStream)
nats stream ls

# 3. Describe stream for detailed info
nats stream info YOUR-STREAM

# 4. List consumers for a stream
nats consumer ls YOUR-STREAM

# 5. Describe consumer
nats consumer info YOUR-STREAM YOUR-CONSUMER

# 6. Check stream storage usage
nats stream info YOUR-STREAM | grep -i storage

# 7. View recent messages on stream
nats stream get YOUR-STREAM --last 10

# 8. Check for message backlog
nats stream info YOUR-STREAM | grep -A2 "State:"

# 9. List subjects with message counts
nats stream report

# 10. Check server logs
journalctl -u nats -f | grep -i error

# 11. Delete old messages to free space (caution)
nats stream purge YOUR-STREAM --before 2024-01-01

# 12. Export stream data
nats stream export YOUR-STREAM /tmp/stream-export

BAD: Not checking consumeracknowledgment settings

# ❌ BAD — may miss why messages aren't being processed
nats consumer info YOUR-STREAM YOUR-CONSUMER
# Without checking AckPolicy, you won't know if messages need explicit ack

GOOD: Verify consumer acknowledgment settings

# ✅ GOOD — check ack policy and pending messages
nats consumer info YOUR-STREAM YOUR-CONSUMER | grep -E "(AckPolicy|NumPending|NumAckPending)"
# If NumAckPending > 0, messages waiting for ack — consumer may be stuck

Pattern 5: Message Backlog Analysis

Problem: Message backlogs growing across multiple queue types, need to diagnose root cause.

# Kafka: Check consumer group lag trends
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group your-group | awk '{print $5}' | tail -20

# RabbitMQ: Calculate queue growth rate
rabbitmqctl list_queues name messages | awk '{sum+=$2} END {print "Total messages:", sum}'

# SQS: Check approximate message counts over time
aws sqs get-queue-attributes --queue-url URL --attribute-names ApproximateNumberOfMessages --query 'Attributes.ApproximateNumberOfMessages'

# NATS: Check stream message count
nats stream info YOUR-STREAM | grep "Messages:" | awk '{print $2}'

# Cross-queue comparison script
cat <<'EOF' > /tmp/queue-diagnostic.sh
#!/bin/bash
echo "=== Kafka ==="
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group $1 2>/dev/null | grep -v "GROUP" | awk '{sum+=$5} END {print "Total Lag:", sum}'

echo "=== RabbitMQ ==="
rabbitmqctl list_queues name messages 2>/dev/null | tail -n +2 | awk '{sum+=$2} END {print "Total Messages:", sum}'

echo "=== SQS ==="
aws sqs get-queue-attributes --queue-url "$QUEUE_URL" --attribute-names ApproximateNumberOfMessages 2>/dev/null | grep -o '[0-9]*'

echo "=== NATS ==="
nats stream info "$STREAM" 2>/dev/null | grep "Messages:" | awk '{print $2}'
EOF

# Run with: bash /tmp/queue-diagnostic.sh consumer-group-name

BAD: Scaling producers to fix consumer backlog

# ❌ BAD — makes backlog worse by adding more messages
kafka-console-producer --broker-list localhost:9092 --topic your-topic < large-file.txt
# Adding messages to already backlogged queue makes problem worse

GOOD: Scale consumers to match processing capacity

# ✅ GOOD — increase consumer parallelism
kafka-console-consumer --bootstrap-server localhost:9092 --topic your-topic --group your-group --num-consumers 4
# Or deploy more consumer instances with same group ID

Pattern 6: Dead Letter Queue Handling

Problem: DLQ filling up with failed messages, need to analyze and process.

# Kafka: Check dead letter topic
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group dlq-consumer-group

# Kafka: Process DLQ messages
kafka-console-consumer --bootstrap-server localhost:9092 --topic your-topic.DLQ --from-beginning --max-messages 100 | while read msg; do
  echo "Processing: $msg"
  # Add your retry logic here
done

# RabbitMQ: List DLQ
rabbitmqctl list_queues name messages | grep -E "(dlq|DLQ|dead)"

# RabbitMQ: Reprocess DLQ messages
rabbitmqadmin get queue=your-queue.DLQ count=10 requeue=true

# SQS: Check DLQ configuration
aws sqs get-queue-attributes --queue-url https://sqs.region.amazonaws.com/account/main-queue --attribute-names RedrivePolicy

# SQS: Process DLQ
aws sqs receive-message --queue-url https://sqs.region.amazonaws.com/account/queue-DLQ --max-number-of-messages 10 --attribute-names All | jq -r '.Messages[].Body' | while read msg; do
  echo "Retrying: $msg"
  # Add your retry logic
  aws sqs delete-message --queue-url URL --receipt-handle RECEIPT-HANDLE
done

# NATS: Check stream for failed messages
nats consumer info YOUR-STREAM YOUR-CONSUMER | grep -E "(NumPending|NumRedelivered)"

# Clean up DLQ after processing (caution!)
rabbitmqctl purge_queue your-queue.DLQ

BAD: Ignoring DLQ error patterns

# ❌ BAD — just purging without analysis
rabbitmqctl purge_queue your-queue.DLQ
# You lose valuable error information this way

GOOD: Analyze DLQ before clearing

# ✅ GOOD — extract error patterns first
rabbitmqctl list_queues name messages | grep -E "(dlq|DLQ)" | while read line; do
  queue=$(echo $line | awk '{print $1}')
  count=$(echo $line | awk '{print $2}')
  echo "DLQ: $queue has $count messages"
  # Process each DLQ before clearing
done

Constraints

MUST DO

• Check cluster connectivity before investigating individual queues (Early Exit)
• Verify consumer group status and lag metrics for all queue types
• Inspect dead letter queues for error pattern analysis
• Validate visibility timeout settings against actual processing duration
• Analyze backlog trends before making scaling decisions
• Confirm network connectivity to queue brokers
• Review broker logs for warnings and errors before application changes
• Test with sample messages after applying fixes

MUST NOT DO

• Scale producers when backlogs exist — only scale consumers
• Disable DLQ processing "temporarily" — this hides failures
• Clear queues without first analyzing message content
• Change visibility timeout without understanding processing duration
• Ignore consumer rebalance events — they indicate instability
• Run purge commands without backup in production environments
• Assume invisible messages are being processed — verify with metrics

TL;DR for Code Generation

• Use guard clauses — return early on invalid queue states
• Return simple types (dict, str, int, bool, list) - avoid complex nested objects
• Handle null/empty cases explicitly at function top (Early Exit)
• Never mutate input parameters — return new dicts/objects
• Fail fast with descriptive errors — don't try to "patch" bad data
• Include timing and latency metadata in all return values
• Reference queue-specific error codes and metrics

Output Template

When applying this skill, produce:

1. Cluster Health Status — Summary of broker connectivity and cluster state
2. Affected Queue List — Queue names with current message counts and consumer status
3. Lag Analysis — Consumer group lag or unacknowledged message counts
4. DLQ Summary — Dead letter queue status and error patterns identified
5. Root Cause — Identified cause (visibility timeout, consumer failure, scaling mismatch)
6. Resolution Steps — Specific commands to apply fixes
7. Verification Steps — Commands to confirm fixes worked
8. Prevention Measures — Configuration changes to avoid recurrence

Related Skills

References

• Apache Kafka Documentation
• RabbitMQ Management Guide
• AWS SQS Documentation
• NATS Documentation
• Kafka CLI Tools Reference
• RabbitMQ CLI Tools
• AWS CLI SQS Commands
• NATS CLI Reference

Quick Reference Commands

Kafka Quick Commands

kafka-topics --bootstrap-server localhost:9092 --list
kafka-consumer-groups --bootstrap-server localhost:9092 --list
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group GROUP-NAME
kafka-console-consumer --bootstrap-server localhost:9092 --topic TOPIC-NAME --from-beginning --max-messages 10

RabbitMQ Quick Commands

rabbitmqctl status
rabbitmqctl list_queues name messages messages_unacknowledged consumers
rabbitmqctl list_consumers
rabbitmqctl purge_queue QUEUE-NAME

SQS Quick Commands

aws sqs list-queues
aws sqs get-queue-attributes --queue-url URL --attribute-names All
aws sqs receive-message --queue-url URL --max-number-of-messages 10
aws sqs set-queue-attributes --queue-url URL --attributes '{"VisibilityTimeout": "300"}'

NATS Quick Commands

nats stream ls
nats stream info STREAM-NAME
nats consumer ls STREAM-NAME
nats consumer info STREAM-NAME CONSUMER-NAME
nats stream get STREAM-NAME --last 10

This skill provides comprehensive message queue troubleshooting for production systems. Always test diagnostic commands in non-production environments first.