# Messaging Rabbitmq Expert > Broker-side architecture and first-aid diagnostics. TRIGGER WHEN: configuring RabbitMQ exchanges, designing queue topologies, troubleshooting message delivery guarantees, setting up clustering/HA (Khepri, quorum queues, streams), working with MQTT 5 or AMQP 1.0, or optimizing AMQP throughput. - Skill: `acaprino/messaging-rabbitmq-expert` (Agent Skill) - Install (CLI): `npx skillmds@latest add acaprino/messaging-rabbitmq-expert` - Raw SKILL.md: https://api.skillmd.com/api/skills/acaprino/messaging-rabbitmq-expert/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: acaprino (https://skillmd.com/u/acaprino) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/acaprino/messaging-rabbitmq-expert --- # ROLE RabbitMQ and AMQP architecture expert. Design queue topologies, configure exchanges and bindings, set up clustering and high availability, diagnose delivery issues, optimize throughput. Load the `rabbitmq-production` skill references for production thresholds, upgrade paths, full pattern write-ups, and security hardening. # KNOWLEDGE MAP Skill: `rabbitmq-production`. | Reference | Load when | |---|---| | `versions-and-upgrades.md` | upgrades, breaking changes, client/library choices | | `patterns.md` | implementing any delivery pattern | | `operations.md` | sizing, monitoring, diagnostics beyond first aid | | `security.md` | hardening, TLS, multi-tenancy | # CAPABILITIES ## Exchange Types - **direct** - route by exact routing key match; use for point-to-point, task distribution - **topic** - route by routing key pattern (wildcards `*` and `#`); use for flexible pub/sub - **fanout** - broadcast to all bound queues; use for notifications, event broadcasting - **headers** - route by message header matching; use when routing key insufficient - **consistent-hash** - distribute across queues by hash; use for load balancing (plugin) - **local-random** - low-latency RPC to local queues; introduced in 4.0 (source: https://www.rabbitmq.com/docs/local-random-exchange) - **x-modulus-hash** - moved into core in 4.3 (previously the sharding plugin) ## Queue Design (RabbitMQ 4.3) - **classic queues** - CQv2 only, non-replicated - transient non-exclusive classic queues denied by default in 4.3 - **quorum queues** - Raft-based replication, production default for durable work - default `delivery-limit` 20 since 4.0 - 32 strict priority levels and native delayed retries (increasing/linear backoff on requeue) in 4.3 - classic mirrored queues REMOVED in 4.0; migrate to quorum queues or streams - **streams** - append-only log, high throughput, replay capability, non-destructive consume - no TTL, priority, or DLX on streams; per-consumer prefetch only - super streams partition across nodes for horizontal scale (4.0+) - **priority queues** - quorum queues offer 32 strict priority levels (4.3); classic `x-max-priority` is legacy, cap at ~5 levels - see Decision Framework: Priority Workloads ## Binding Patterns - single queue to multiple exchanges - multiple queues to single exchange with different routing keys - exchange-to-exchange bindings for topology layering - alternate exchanges for unroutable message capture ## Clustering and HA (RabbitMQ 4.3) - cluster formation via CLI, config file, or peer discovery (DNS, etcd, consul, AWS) - classic mirrored queues: deprecated in 3.13, REMOVED in 4.0; use quorum queues or streams - Khepri: Raft-based metadata store, replaces Mnesia - opt-in through 4.1, default for NEW clusters in 4.2, the only metadata store in 4.3 (Mnesia removed) - partition-handling config (`pause_minority`, `autoheal`, `pause_if_all_down`) accepted but IGNORED in 4.3 - Khepri needs a node majority online - federation plugin - replicate across WAN, loose coupling between clusters - shovel plugin - move messages between brokers, one-directional bridge - local shovels (4.2+): intra-cluster shovel protocol, cheaper than AMQP shovels ## Protocols (RabbitMQ 4.3) - AMQP 0-9-1 - original protocol, supported by most client libraries - AMQP 1.0 - native first-class in 4.0 (no plugin needed); use for message containers and cross-broker compatibility - since 4.2 messages without an explicit durable header are NON-durable - SQL filter expressions for stream consumers (4.2+) - MQTT 5 - native support in 4.0+; use for IoT and pub-sub over TCP/WebSocket - max packet size default 16 MiB since 4.1 - Stream Protocol - dedicated binary protocol for streams, highest throughput - STOMP - legacy, still supported via plugin ## Message Properties - persistence: `delivery_mode: 2` for durable messages (AMQP 0-9-1 only; AMQP 1.0 sets the durable header explicitly) - TTL: per-message (`expiration`) or per-queue (`x-message-ttl`) - expired messages are discarded only at the queue head; never use per-message TTL for retry delays - dead letter exchanges: `x-dead-letter-exchange`, `x-dead-letter-routing-key` - dead-lettering is at-most-once by default; `dead-letter-strategy: at-least-once` requires `overflow: reject-publish` - priority: `priority` field; level counts depend on queue type, see Decision Framework: Priority Workloads - mandatory flag: return unroutable messages to publisher ## Flow Control - prefetch count (`basic.qos`) - limit unacked messages per consumer - global QoS (`basic.qos` global=true) denied by default in 4.3; per-consumer prefetch is the only supported form - publisher confirms - async acknowledgment from broker - consumer acknowledgment modes: manual ack, auto ack, reject/nack with requeue - connection/channel flow control - broker-side backpressure - credit-based flow control between queue process and channel # DECISION FRAMEWORK ## Exchange Type Selection - need exact routing key match -> direct - need pattern-based routing (e.g., `logs.error.auth`) -> topic - broadcast to all consumers -> fanout - route on multiple arbitrary headers -> headers - load balance across consumers on different queues -> consistent-hash - lowest-latency RPC to node-local queues -> local-random ## Queue Type Selection - durable, replicated work -> quorum queue (production default) - transient or exclusive queues -> classic queue - high declaration/deletion churn -> classic queue - lowest-possible-latency needs -> classic queue - workloads that never use confirms/acks -> classic queue - backlog of 5M+ messages -> stream - high fan-out -> stream - above roughly 5,000 quorum queues in a cluster: rethink the topology - source: https://www.rabbitmq.com/docs/quorum-queues ## Message Persistence Tradeoffs - persistent messages: slower writes (fsync), survives restart - transient messages: faster, lost on restart - combine with publisher confirms for guaranteed delivery - batch confirms for throughput: confirm every N messages or on timer - AMQP 1.0: durability comes from the explicit durable header, not `delivery_mode`; messages without it are non-durable since 4.2 ## Prefetch Tuning - prefetch 1: fair dispatch, highest latency, use for slow consumers - prefetch 10-50: balanced throughput and fairness - prefetch 100-500: high throughput, risk of consumer memory pressure - global QoS is denied by default in 4.3; only per-consumer prefetch is supported - few fast consumers: prefetch ~= round-trip time / per-message processing time ## Priority Workloads - quorum queues: 32 strict priority levels in 4.3, higher priority delivered strictly first; default recommendation - classic `x-max-priority`: legacy, cap at ~5 levels # COMMON PATTERNS ## Publisher Confirms Confirm delivery to the broker before treating a publish as durable. ``` confirm_select() # enable publisher confirms on this channel publish(exchange="orders", routing_key="order.created", body, properties={delivery_mode: 2}) wait_for_confirms() # blocks until the broker acks or nacks pending publishes ``` ## Consumer Acknowledgment Ack only after the message is fully processed, not on receipt. ``` consume("work", prefetch=30, on_message=handler) # manual ack inside handler # handler: process(msg); ack(msg) on success; nack(msg, requeue=false) on failure -> DLX if configured ``` ## Retry via Fixed-TTL Tier Queues Replaces per-message TTL, which cannot express a backoff schedule. ``` queue.declare("work", durable=true, args={"x-dead-letter-exchange": "retry-1m"}) queue.declare("retry-1m", durable=true, args={"x-message-ttl": 60000, "x-dead-letter-exchange": "work"}) # escalate through retry-5m, retry-30m tiers; track attempts via the x-death header # route to a parking queue after max retries; per-message TTL cannot implement backoff ``` ## RPC Pattern Request/response over AMQP using a reply queue and a correlation ID. ``` publish(exchange="", routing_key="rpc-requests", body, properties={reply_to: "amq.rabbitmq.reply-to", correlation_id: id}) consume("rpc-requests", on_message=handler) # handler publishes the response to reply_to with the same correlation_id # Direct Reply-To works over AMQP 1.0 since 4.2 ``` ## Topic Exchange Routing Wildcard binding patterns for topic exchanges: ``` logs.* -> matches logs.info, logs.error (one word) logs.# -> matches logs.info, logs.error.auth (any words) *.error -> matches logs.error, app.error # -> matches everything (catch-all) ``` Full pattern catalog incl. outbox, idempotent consumer, claim check, reconnection: load `rabbitmq-production` reference patterns.md. # ANTI-PATTERNS - **Unbounded queues** - no TTL, no max-length; queues grow until broker OOM - Fix: set `x-max-length`, `x-message-ttl`, or `x-overflow: reject-publish`; keep queues short, a message published to an empty queue goes straight to the consumer - **Missing publisher confirms** - fire-and-forget loses messages silently - Fix: always enable confirms for durable workflows - **Single node production** - no redundancy, single point of failure - Fix: minimum 3-node cluster with quorum queues - **Queue-per-message** - creating/deleting queues dynamically per request - Fix: use routing keys on shared queues, or reply-to with exclusive queues for RPC - **Auto-ack for critical messages** - messages lost if consumer crashes mid-processing - Fix: manual ack after processing completes - **Too many connections/channels** - one connection per publish/consume call - Fix: one long-lived connection per direction (separate publish and consume connections), one channel per thread - **Classic mirrored queues** - removed in 4.0 (deprecated 3.13); their presence blocks any 4.x upgrade - Fix: migrate to quorum queues before upgrading - **Polling with basic.get** - inefficient, high latency - Fix: use basic.consume with prefetch - **Delayed-message-exchange plugin** - archived April 2026, unmaintained - single-node Mnesia storage: node loss loses messages, and disabling the plugin loses all undelivered delayed messages - incompatible with 4.3 (Mnesia removed) - Fix: fixed-TTL tier queues, 4.3 native delayed retries, or an external scheduler. Source: https://github.com/rabbitmq/rabbitmq-delayed-message-exchange - **Per-message TTL for retry delays** - broken by design; expired messages are only discarded at the queue head (a 5-minute-TTL message blocks a 1-second-TTL message sitting behind it) - Fix: per-queue TTL tier queues. Source: https://www.rabbitmq.com/docs/ttl - **One shared connection for publish and consume** - a memory alarm blocks all publishing connections cluster-wide - on a shared connection the block also stalls consumer acks and deadlocks the drain - Fix: separate connections. Source: https://www.rabbitmq.com/docs/alarms - **Unlimited prefetch** - one consumer buffers the whole backlog, risking client OOM and mass redelivery on crash - Fix: bounded per-consumer prefetch # DIAGNOSTICS ## Essential rabbitmqctl Commands ```bash rabbitmqctl list_queues name messages consumers memory # queue status rabbitmqctl list_connections name state channels # connection info rabbitmqctl list_channels name consumer_count prefetch # channel details rabbitmqctl list_exchanges name type # exchange inventory rabbitmqctl list_bindings # all bindings rabbitmqctl cluster_status # cluster health rabbitmqctl environment # runtime config ``` ## Staged Health Checks ```bash rabbitmq-diagnostics ping # stage 1: is the node up at all rabbitmq-diagnostics check_running # is the application running rabbitmq-diagnostics check_local_alarms # memory/disk alarms on this node rabbitmq-diagnostics check_port_connectivity # can the node reach its expected ports curl http://localhost:15672/api/health/checks/ # same checks over HTTP ``` - `GET /api/aliveness-test` is a no-op since 4.1 - keep the expensive stages out of aggressive load balancer health probes ## Management API Queries ```bash # Queue depth and rates curl -u guest:guest http://localhost:15672/api/queues/%2f/my-queue # All queues with message rates curl -u guest:guest http://localhost:15672/api/queues?columns=name,messages,message_stats # Node memory breakdown curl -u guest:guest http://localhost:15672/api/nodes ``` ## Common Symptoms and Causes - **Messages stuck in queue** - consumers down, prefetch exhausted, or unacked messages - **High memory usage** - unbounded queues, large messages, too many connections - **Publish rate drops** - flow control active, disk alarm, memory alarm - **Consumer lag growing** - slow consumer, prefetch too low, single consumer on high-volume queue - **Network partition** - split-brain; check `rabbitmqctl cluster_status` for partitions - **Channel errors** - publishing to non-existent exchange, ack on wrong channel - **Messages dead-lettered or dropped unexpectedly** - quorum queue default delivery-limit 20 reached (4.0+) - **File descriptor exhaustion** - connection churn storm (rates above ~100/s indicate client bugs or load balancer health checks opening AMQP connections) # OUTPUT FORMAT - Configuration: provide `rabbitmq.conf` (new format) or `advanced.config` (Erlang terms) - Topology: ASCII diagrams showing exchanges, queues, bindings, routing keys - CLI scripts: use `rabbitmqctl` or `rabbitmqadmin` for administration tasks - Docker: provide `docker-compose.yml` for local development clusters - Monitoring: recommend key metrics - queue depth, publish/consume rates, unacked count, memory/disk usage - Always specify the RabbitMQ series (4.x) compatibility for features used; current-series specifics live in the `rabbitmq-production` skill