RabbitMQ Architecture Designer

RabbitMQ Architecture Designer

Purpose & When-To-Use

Trigger conditions:

• You need to design a RabbitMQ topology with exchanges, queues, and routing patterns
• You need to select queue types (classic, quorum, streams) based on durability and replication needs
• You need to configure publisher confirms or consumer acknowledgments for reliability
• You need to set up clustering for high availability with quorum queue replication
• You need to implement dead letter exchange (DLX) error handling or retry patterns
• You need to optimize consumer prefetch or concurrent processing

Complements:

• integration-messagequeue-designer: For generic message queue pattern selection (RabbitMQ vs Kafka vs SQS)
• messaging-kafka-architect: For Kafka-specific event streaming architectures
• microservices-pattern-architect: For saga, CQRS, event sourcing patterns that use RabbitMQ

Out of scope:

• RabbitMQ installation and OS-level configuration (use infrastructure automation)
• Monitoring and alerting setup (use observability-stack-configurator)
• Client library integration code (use language-specific AMQP client docs)
• Long-term message retention (use Kafka Streams or database for archival)

Pre-Checks

Time normalization:

• Compute NOW_ET using NIST/time.gov semantics (America/New_York, ISO-8601)
• Use NOW_ET for all access dates in citations

Verify inputs:

• ✅ Required: At least one message flow definition (publisher → exchange → queue → consumer)
• ✅ Required: RabbitMQ version specified (recommend 4.2+ for Khepri metadata store and quorum queue enhancements)
• ⚠️ Optional: Exchange type preferences (default to topic for flexibility)
• ⚠️ Optional: Queue type (classic, quorum, streams) - default to quorum for durability
• ⚠️ Optional: Clustering requirements (node count, replication factor)
• ⚠️ Optional: Error handling strategy (DLX, retry backoff, TTL)

Validate requirements:

• If high availability needed → quorum queues (since RabbitMQ 3.8, replicated via Raft)
• If ordering required → single active consumer (SAC) or stream queues
• If priorities needed → quorum queues support 2 priorities (high/normal) in RabbitMQ 4.0+
• If broadcasting → fanout exchange
• If complex routing → topic exchange with wildcards

Source freshness:

• RabbitMQ 4.2 latest (2025, Khepri metadata store, stream filters) (accessed NOW_ET)
• Quorum queues introduced in 3.8, enhanced in 4.0 (priorities, consumer priority for SAC)
• Classic mirrored queues removed in 4.0 (replaced by quorum queues)

Abort if:

• No message flow specified → EMIT TODO: "Define at least one publisher → exchange → queue → consumer flow"
• Queue type unclear → EMIT TODO: "Specify queue requirements: durability (classic/quorum), replication (quorum), or high throughput (streams)"
• Clustering without quorum queues → EMIT TODO: "Use quorum queues for replicated, highly available queues (classic queues are single-node in RabbitMQ 4.x)"

Procedure

T1: Basic RabbitMQ Topology (≤2k tokens, 80% use case)

Scenario: Single exchange, single queue, direct routing, no clustering, basic error handling.

Steps:

1. Exchange Type Selection:

   • Direct: Exact routing key match (e.g., order.created → order-processing-queue)
   • Topic: Pattern matching with * (1 word) and # (0+ words) (e.g., audit.events.# matches audit.events.users.signup)
   • Fanout: Broadcast to all queues (ignore routing key)
   • T1 recommendation: Use topic exchange for flexibility even if only using exact routing initially

2. Queue Type Selection:

   • Classic: Single node, non-replicated (use only for dev/test)
   • Quorum: Replicated (Raft consensus), durable, data safety (use for production)
   • Streams: High throughput, append-only log (use for event streaming)
   • T1 recommendation: Use quorum queue with x-queue-type=quorum argument

3. Publisher Configuration:

   • Enable publisher confirms for reliability (wait for broker acknowledgment)
   • Set delivery mode = 2 for persistent messages (survive broker restart)
   • T1 recommendation: Use streaming confirms (handle confirms as they arrive)

4. Consumer Configuration:

   • Use manual acknowledgments (ack after successful processing)
   • Set prefetch count = 10 (balance between throughput and backpressure)
   • T1 recommendation: Ack after processing, nack+requeue on transient errors

5. Basic Topology:

   • 1 topic exchange (events)
   • 1 quorum queue (order-processing-queue)
   • 1 binding (order.created → order-processing-queue)
   • Publisher → events exchange with routing key order.created
   • Consumer → order-processing-queue with manual ack + prefetch=10

Output:

• Topology diagram: 1 exchange, 1 queue, 1 binding
• Publisher config: confirms enabled, persistent messages
• Consumer config: manual ack, prefetch=10

Token budget: ≤2000 tokens

T2: Multi-Exchange Routing + DLX Error Handling (≤6k tokens)

Scenario: Multiple exchanges with complex routing, dead letter exchange for errors, quorum queues, retry with backoff.

Steps:

1. Multi-Exchange Topology:

   Pattern: Separate exchanges for different message types or bounded contexts

   • Example:
     • orders-exchange (topic) → routes to order-processing-queue, order-audit-queue
     • payments-exchange (topic) → routes to payment-processing-queue
     • notifications-exchange (fanout) → broadcasts to all notification queues

2. Topic Exchange Routing Patterns:

   Wildcards:

   • * matches exactly one word (e.g., order.*.created matches order.online.created but not order.created)
   • # matches zero or more words (e.g., audit.# matches audit.users, audit.users.signup, audit)

   Example bindings:

   • order.created → order-processing-queue (exact match)
   • order.# → order-audit-queue (all order events)
   • payment.processed → payment-processing-queue
   • notification.* → notification-email-queue, notification-sms-queue (broadcast via topic)

3. Dead Letter Exchange (DLX) Setup:

   Use cases:

   • Handle messages rejected by consumers (nack without requeue)
   • Handle messages exceeding TTL (time-to-live)
   • Handle messages exceeding delivery limit (quorum queues default limit=20)

   Configuration via policy (recommended):

   {
     "pattern": "order-processing-queue",
     "definition": {
       "dead-letter-exchange": "dlx-exchange",
       "dead-letter-routing-key": "order.processing.failed",
       "message-ttl": 86400000,
       "delivery-limit": 20
     }
   }
   

   DLX topology:

   • Main queue: order-processing-queue (quorum)
   • Dead letter exchange: dlx-exchange (topic)
   • Dead letter queue: dlx-order-processing-queue (quorum, for manual inspection)
   • Binding: order.processing.failed → dlx-order-processing-queue

4. Retry with Backoff Pattern:

   Pattern: Use TTL + DLX to implement delayed retries

   • Step 1: Consumer nacks message without requeue → DLX routes to retry-queue-5s (TTL=5s)
   • Step 2: After 5s, message expires → routes back to main queue via DLX
   • Step 3: Repeated failures trigger delivery limit → routes to final DLX for manual handling

   Example:

   • Main queue: order-processing-queue
   • Retry queue 1: retry-order-5s (TTL=5s, DLX=orders-exchange)
   • Retry queue 2: retry-order-30s (TTL=30s, DLX=orders-exchange)
   • Final DLX: dlx-order-processing-queue (manual inspection)

5. Quorum Queue Configuration:

   Arguments:

   • x-queue-type=quorum (replicated queue)
   • x-quorum-initial-group-size=3 (replication factor, odd number for Raft consensus)
   • x-delivery-limit=20 (max redeliveries before DLX, default in RabbitMQ 4.0+)
   • x-max-priority=2 (RabbitMQ 4.0+ supports exactly 2 priorities: normal and high)

   Publisher priority:

   • Publish with priority=5 (high priority, delivered 2:1 ratio vs normal)
   • Publish with priority=0 or no priority (normal priority)

6. Consumer Acknowledgment Strategies:

   Manual ack (recommended):

   • Process message → basic.ack (remove from queue)
   • Transient error (network timeout) → basic.nack + requeue=true (redelivery)
   • Permanent error (invalid data) → basic.nack + requeue=false (send to DLX)

   Prefetch tuning:

   • Low prefetch (1-10): Better fairness, lower throughput
   • High prefetch (50-100): Higher throughput, risk of consumer overload
   • Recommendation: Start with prefetch=10, tune based on processing time and consumer count

Output:

• Multi-exchange topology (orders, payments, notifications)
• Topic routing patterns with wildcards
• DLX error handling with retry backoff
• Quorum queue configuration
• Publisher/consumer config (confirms, acks, prefetch)

Token budget: ≤6000 tokens

T3: Clustering + Streams + Advanced Patterns (≤12k tokens)

Scenario: Multi-node cluster with quorum queue replication, stream queues for high throughput, federation for multi-DC.

Steps:

1. Clustering Topology:

   Best practices:

   • Odd number of nodes: 3, 5, or 7 nodes (Raft consensus requires majority)
   • Equal peers: All nodes are equal (no leader/follower at cluster level, but quorum queues use Raft leader election)
   • Network requirements: Nodes must resolve hostnames, ports 4369 (epmd), 25672 (inter-node), 5672 (AMQP) open
   • Avoid 2-node clusters: No clear majority during network partitions

   Example 3-node cluster:

   • Node 1: rabbit@node1.example.com
   • Node 2: rabbit@node2.example.com
   • Node 3: rabbit@node3.example.com
   • Erlang cookie: same on all nodes (authentication)

   Quorum queue replication:

   • Quorum queues replicate across 3 nodes (configurable via x-quorum-initial-group-size)
   • Raft leader elected automatically (handles writes)
   • Followers replicate data (handle reads if leader down)
   • Survives minority node failures (e.g., 1 node down in 3-node cluster)

2. Stream Queues for High Throughput:

   Use case: Event streaming, audit logs, high-volume data ingestion (millions of messages/sec)

   Characteristics:

   • Append-only log (like Kafka topics)
   • Multiple consumers can read from same offset
   • Retention based on size or time (not per-consumer)
   • RabbitMQ 4.2: SQL filter expressions (4M+ msg/sec filtering with Bloom filters)

   Configuration:

   {
     "x-queue-type": "stream",
     "x-max-age": "7D",
     "x-stream-max-segment-size-bytes": 500000000
   }
   

   Consumer offset tracking:

   • Consumer specifies offset: first, last, next, or timestamp
   • Offset stored server-side (like Kafka consumer groups)

3. Consistent Hashing Exchange (Plugin):

   Use case: Shard messages across multiple queues for horizontal scaling

   Pattern:

   • Consistent hashing exchange routes based on routing key hash
   • Messages with same routing key always go to same queue (ordering guarantee)
   • Add/remove queues with minimal redistribution

   Example:

   • Exchange: sharded-orders (type=x-consistent-hash)
   • Queues: orders-shard-0, orders-shard-1, orders-shard-2
   • Routing key: user-123 → always routes to same shard

4. Federation for Multi-DC:

   Use case: Replicate messages across datacenters without clustering (clusters require low-latency networks)

   Pattern:

   • Upstream (DC1): orders-exchange
   • Downstream (DC2): orders-exchange-federated (receives messages from DC1)
   • Federation link: DC2 pulls messages from DC1 orders-exchange

   Benefits:

   • Survives WAN latency and network partitions (unlike clustering)
   • Independent RabbitMQ clusters in each DC
   • Messages flow one-way (upstream → downstream)

5. Advanced Publisher Patterns:

   Transactional publishing (avoid, heavyweight):

   • AMQP transactions (tx.select, tx.commit) → very slow, blocks channel
   • Use publisher confirms instead (asynchronous, higher throughput)

   Batch publishing:

   • Publish multiple messages, then wait for confirms in batch
   • Higher throughput than individual confirms
   • Risk: larger batch = longer recovery time on failure

6. Single Active Consumer (SAC) for Ordering:

   Use case: Ensure messages processed in order by allowing only one consumer at a time

   Configuration:

   • Queue argument: x-single-active-consumer=true
   • RabbitMQ selects one consumer as active, others wait
   • Automatic failover to standby consumer if active consumer dies
   • RabbitMQ 4.0+: Consumer priority for SAC (higher priority consumers selected first)

7. Message Priority in Quorum Queues:

   RabbitMQ 4.0+ feature:

   • Quorum queues support exactly 2 priorities: high and normal
   • No upfront declaration needed (unlike classic queues)
   • Consumers receive 2:1 ratio of high to normal priority messages (avoid starvation)
   • Publish with priority=5 (high) or priority=0/unset (normal)

Output:

• 3-node cluster topology with quorum queue replication
• Stream queue configuration for high-throughput use cases
• Consistent hashing exchange for sharding
• Federation setup for multi-DC replication
• SAC and message priority patterns

Token budget: ≤12000 tokens

Decision Rules

Exchange type selection:

• Direct: Exact routing, one-to-one (e.g., task queues, RPC)
• Topic: Pattern matching, one-to-many with hierarchical routing (e.g., event bus, audit logs)
• Fanout: Broadcast, one-to-all (e.g., notifications, cache invalidation)
• Headers: Route by message headers (rare, use topic instead)

Queue type selection:

• Classic: Dev/test only (single node, non-replicated in RabbitMQ 4.x)
• Quorum: Production (replicated, durable, Raft consensus, survives node failures)
• Streams: High throughput + retention (append-only, multi-consumer reads, event streaming)

Clustering decisions:

• Single node: Dev/test, <1000 msg/sec
• 3-node cluster: Production, high availability, survives 1 node failure
• 5-node cluster: Mission-critical, survives 2 node failures
• 7+ node cluster: Rare (Raft consensus overhead increases, consider federation instead)

Prefetch tuning:

• 1-10: Low throughput, fair distribution, consumer processing time >100ms
• 10-50: Medium throughput, balanced, consumer processing time 10-100ms
• 50-100: High throughput, consumer processing time <10ms

Error handling strategy:

• Transient errors: nack + requeue=true (network timeout, downstream unavailable)
• Permanent errors: nack + requeue=false → DLX (invalid data, schema mismatch)
• Retry with backoff: DLX → TTL queue → re-route to main queue after delay
• Poison messages: Delivery limit (default=20) → DLX for manual inspection

Abort conditions:

• Quorum queue replication factor >cluster size → reduce to match node count
• Prefetch >1000 → risk of consumer memory exhaustion
• Classic queues in production → migrate to quorum queues for durability

Output Contract

Topology schema:

exchanges:
  - name: <exchange_name>
    type: direct|topic|fanout|headers
    durable: true|false
    auto_delete: true|false

queues:
  - name: <queue_name>
    type: classic|quorum|stream
    durable: true|false
    arguments:
      x-queue-type: quorum
      x-quorum-initial-group-size: 3
      x-delivery-limit: 20
      x-max-priority: 2  # RabbitMQ 4.0+ only
      x-single-active-consumer: true|false

bindings:
  - exchange: <exchange_name>
    queue: <queue_name>
    routing_key: <pattern>  # e.g., order.created, order.#, *

policies:
  - name: <policy_name>
    pattern: <queue_regex>
    definition:
      dead-letter-exchange: <dlx_exchange>
      dead-letter-routing-key: <dlx_routing_key>
      message-ttl: <milliseconds>
      delivery-limit: 20

Publisher config:

# Publisher confirms
channel.confirm_delivery()

# Persistent messages
channel.basic_publish(
    exchange='orders-exchange',
    routing_key='order.created',
    body=message,
    properties=pika.BasicProperties(
        delivery_mode=2,  # persistent
        priority=5        # high priority (RabbitMQ 4.0+)
    )
)

Consumer config:

# Manual ack + prefetch
channel.basic_qos(prefetch_count=10)

def callback(ch, method, properties, body):
    try:
        process(body)
        ch.basic_ack(delivery_tag=method.delivery_tag)
    except TransientError:
        ch.basic_nack(delivery_tag=method.delivery_tag, requeue=True)
    except PermanentError:
        ch.basic_nack(delivery_tag=method.delivery_tag, requeue=False)  # → DLX

channel.basic_consume(queue='order-processing-queue',

Required fields:

• Topology: exchanges[], queues[], bindings[]
• Exchange: name, type
• Queue: name, type (classic/quorum/stream)
• Binding: exchange, queue, routing_key

Examples

Example: E-commerce Order Processing with DLX

Topology:

• Exchange: orders-exchange (topic)
• Queue: order-processing-queue (quorum, x-quorum-initial-group-size=3)
• DLX: dlx-exchange (topic)
• DLX Queue: dlx-order-processing-queue (quorum, manual inspection)
• Binding: order.created → order-processing-queue
• DLX Binding: order.processing.failed → dlx-order-processing-queue

Policy (DLX config):

{
  "pattern": "order-processing-queue",
  "definition": {
    "dead-letter-exchange": "dlx-exchange",
    "dead-letter-routing-key": "order.processing.failed",
    "delivery-limit": 20
  }
}

Publisher:

channel.basic_publish(
    exchange='orders-exchange',
    routing_key='order.created',
    body=json.dumps(order),
    properties=pika.BasicProperties(delivery_mode=2)
)

Consumer:

def process_order(ch, method, properties, body):
    try:
        order = json.loads(body)
        # Process order (may fail)
        charge_payment(order)
        ch.basic_ack(delivery_tag=method.delivery_tag)
    except PaymentGatewayDown:  # Transient error
        ch.basic_nack(delivery_tag=method.delivery_tag, requeue=True)
    except InvalidPaymentMethod:  # Permanent error
        ch.basic_nack(delivery_tag=method.delivery_tag, requeue=False)  # → DLX

Quality Gates

Token budgets:

• T1: ≤2000 tokens (single exchange + queue + basic config)
• T2: ≤6000 tokens (multi-exchange + DLX + routing patterns)
• T3: ≤12000 tokens (clustering + streams + federation)

Safety:

• ❌ Never: Hardcode credentials in topology definitions
• ❌ Never: Use classic queues for production (single node, no replication)
• ✅ Always: Enable publisher confirms for reliability
• ✅ Always: Use manual acks for consumers (process then ack)
• ✅ Always: Use quorum queues for durability (replicated, Raft consensus)

Auditability:

• All topology definitions in version control (Git)
• Policies defined via management API or config (not hardcoded queue arguments)
• DLX queues monitored for poison messages
• Consumer ack/nack rates tracked (avoid excessive requeues)

Determinism:

• Same topology definition = same exchange/queue/binding creation
• Quorum queue leader election deterministic (Raft)
• Topic routing deterministic (same routing key → same queue)

Performance:

• Prefetch tuned for consumer processing time (avoid memory exhaustion)
• Quorum queue replication factor ≤ node count
• Stream queues for >10k msg/sec throughput
• Publisher confirms in batches for higher throughput (not individual)

Resources

Official Documentation:

• RabbitMQ 4.1.0 release (Khepri metadata store, quorum queue enhancements): https://www.rabbitmq.com/blog/2025/04/15/rabbitmq-4.1.0-is-released (accessed NOW_ET)
• Quorum queues: https://www.rabbitmq.com/docs/quorum-queues (accessed NOW_ET)
• Exchanges and routing: https://www.rabbitmq.com/docs/exchanges (accessed NOW_ET)
• Clustering: https://www.rabbitmq.com/docs/clustering (accessed NOW_ET)
• Dead letter exchanges: https://www.rabbitmq.com/docs/dlx (accessed NOW_ET)
• Publishers: https://www.rabbitmq.com/docs/publishers (accessed NOW_ET)
• Consumers: https://www.rabbitmq.com/docs/consumers (accessed NOW_ET)

Client Libraries:

• Python: pika (AMQP 0-9-1 client)
• Java: amqp-client (official Java client)
• Node.js: amqplib
• Go: amqp091-go

Related Skills:

• integration-messagequeue-designer: Generic message queue pattern selection
• messaging-kafka-architect: Kafka-specific event streaming
• microservices-pattern-architect: Saga, CQRS, event sourcing with RabbitMQ
• observability-stack-configurator: Monitoring RabbitMQ with Prometheus + Grafana