Skill — Queue Design
When this skill activates
Any task involving message broker selection, queue topology, consumer patterns,
TTL policies, priority queues, dead letter queues, or partition strategies.
Mandatory actions when this skill is active
Before writing any code
- Identify throughput requirements (messages/sec peak and sustained).
- Determine ordering and delivery guarantee requirements.
- Select broker based on the decision matrix.
During implementation
- Configure DLQ for every consumer with retry limits.
- Set TTL on messages to prevent unbounded queue growth.
- Implement consumer health checks and graceful shutdown.
- Use partition keys for ordering-sensitive flows.
After implementation
- Set up queue depth monitoring with alerting thresholds.
- Document topology, routing rules, and consumer ownership.
- Verify DLQ workflow (inspect, fix, replay, discard).
Broker Selection
| Broker |
Strength |
Ordering |
Use For |
| Kafka |
Extreme throughput, replay, retention |
Per-partition |
Event sourcing, stream processing, audit logs |
| RabbitMQ |
Flexible routing, per-message ack |
Per-queue FIFO |
Task queues, RPC, complex routing |
| SQS |
Zero ops, infinite scale, built-in DLQ |
FIFO queues (limited) |
Decoupling services, background jobs |
Queue Topologies
- Point-to-Point: one queue, competing consumers. Each message consumed once.
- Fan-Out: one message to multiple queues/groups. Use for event broadcasting.
- Topic-Based: route by pattern matching. Use for selective subscription.
Consumer Patterns
- Competing Consumers: multiple instances share workload, scale by adding consumers.
- Exclusive Consumer: one active at a time, failover pattern for ordered processing.
TTL and Expiration
- Queue-level TTL: all messages expire after N seconds if unconsumed.
- Per-message TTL: individual expiration. Typical: notifications 5min, business 24h, audit 7d.
- Expired messages route to DLQ or silently drop (configure explicitly).
Priority Queues
- Limited levels (1-10 max, typically 3: high/normal/low).
- Risk: starvation of low-priority. Mitigate with aging or separate queues.
Dead Letter Queue Design
message → consumer → FAIL → retry (3x exponential) → FAIL → DLQ → alert
- Retain original body, headers, error details, retry count.
- Alert on first DLQ arrival. Never auto-replay without root-cause fix.
- Replay tool moves DLQ → original queue in small monitored batches.
Partition Strategies
- Partition key = entity ID ensures per-entity ordering.
- Start with 2x expected consumer count as partition count.
- Monitor for hot partitions (skewed key distribution).
Self-check before task completion
1---2name: queue-design3description: Skill — Queue Design4---56# Skill — Queue Design78## When this skill activates9Any task involving message broker selection, queue topology, consumer patterns,10TTL policies, priority queues, dead letter queues, or partition strategies.1112## Mandatory actions when this skill is active1314### Before writing any code151. Identify throughput requirements (messages/sec peak and sustained).162. Determine ordering and delivery guarantee requirements.173. Select broker based on the decision matrix.1819### During implementation20- Configure DLQ for every consumer with retry limits.21- Set TTL on messages to prevent unbounded queue growth.22- Implement consumer health checks and graceful shutdown.23- Use partition keys for ordering-sensitive flows.2425### After implementation26- Set up queue depth monitoring with alerting thresholds.27- Document topology, routing rules, and consumer ownership.28- Verify DLQ workflow (inspect, fix, replay, discard).2930## Broker Selection3132| Broker | Strength | Ordering | Use For |33|--------|----------|----------|---------|34| Kafka | Extreme throughput, replay, retention | Per-partition | Event sourcing, stream processing, audit logs |35| RabbitMQ | Flexible routing, per-message ack | Per-queue FIFO | Task queues, RPC, complex routing |36| SQS | Zero ops, infinite scale, built-in DLQ | FIFO queues (limited) | Decoupling services, background jobs |3738## Queue Topologies3940- **Point-to-Point**: one queue, competing consumers. Each message consumed once.41- **Fan-Out**: one message to multiple queues/groups. Use for event broadcasting.42- **Topic-Based**: route by pattern matching. Use for selective subscription.4344## Consumer Patterns4546- **Competing Consumers**: multiple instances share workload, scale by adding consumers.47- **Exclusive Consumer**: one active at a time, failover pattern for ordered processing.4849## TTL and Expiration50- Queue-level TTL: all messages expire after N seconds if unconsumed.51- Per-message TTL: individual expiration. Typical: notifications 5min, business 24h, audit 7d.52- Expired messages route to DLQ or silently drop (configure explicitly).5354## Priority Queues55- Limited levels (1-10 max, typically 3: high/normal/low).56- Risk: starvation of low-priority. Mitigate with aging or separate queues.5758## Dead Letter Queue Design5960```61message → consumer → FAIL → retry (3x exponential) → FAIL → DLQ → alert62```6364- Retain original body, headers, error details, retry count.65- Alert on first DLQ arrival. Never auto-replay without root-cause fix.66- Replay tool moves DLQ → original queue in small monitored batches.6768## Partition Strategies69- Partition key = entity ID ensures per-entity ordering.70- Start with 2x expected consumer count as partition count.71- Monitor for hot partitions (skewed key distribution).7273## Self-check before task completion7475- [ ] Is the right broker selected for throughput and routing needs?76- [ ] Are DLQs configured with alerting for every consumer?77- [ ] Is TTL set to prevent unbounded growth?78- [ ] Are ordering guarantees maintained where business logic requires?79- [ ] Is the consumer pattern correct (competing vs exclusive)?80- [ ] Are queue depth and consumer lag monitored?81- [ ] Is the DLQ replay workflow documented?
sairam0424