Message Queue Knowledge Base
Quick reference for message broker operations and advanced messaging patterns. Focuses on broker-level operations — for event-driven patterns, see eda-knowledge.
Broker Comparison
| Feature |
RabbitMQ |
Apache Kafka |
Amazon SQS |
Redis Streams |
| Model |
Message queue |
Event log |
Message queue |
Event log |
| Ordering |
Per-queue FIFO |
Per-partition |
Best-effort (FIFO available) |
Per-stream |
| Retention |
Until consumed |
Time/size-based |
4-14 days |
Memory/size-based |
| Replay |
No (once consumed) |
Yes (offset seek) |
No |
Yes (ID-based) |
| Consumer Groups |
Competing consumers |
Native support |
Not built-in |
Native (XREADGROUP) |
| Throughput |
~50K msg/s |
~1M msg/s |
~3K msg/s per queue |
~100K msg/s |
| Latency |
Sub-millisecond |
Low milliseconds |
10-100ms |
Sub-millisecond |
| Protocol |
AMQP 0-9-1 |
Custom (TCP) |
HTTP/SQS API |
RESP |
| Clustering |
Quorum queues |
Built-in (ZK/KRaft) |
Managed |
Redis Cluster |
| Best for |
Task queues, RPC |
Event streaming, logs |
Serverless, AWS-native |
Lightweight streaming |
Message Delivery Guarantees
| Guarantee |
Description |
Implementation |
Trade-off |
| At-most-once |
Message may be lost |
Fire-and-forget, no ack |
Fastest, data loss possible |
| At-least-once |
Message delivered 1+ times |
Ack after processing |
Requires idempotent consumers |
| Exactly-once |
Message processed exactly once |
Transactional + deduplication |
Slowest, most complex |
Achieving At-Least-Once in PHP
// RabbitMQ: manual acknowledgment
$channel->basic_consume(
queue: 'orders',
no_ack: false, // require explicit ack
callback: function (AMQPMessage $msg) use ($channel): void {
try {
$this->handler->handle(json_decode($msg->getBody(), true));
$channel->basic_ack($msg->getDeliveryTag());
} catch (\Throwable $e) {
$channel->basic_nack($msg->getDeliveryTag(), requeue: true);
}
},
);
Consumer Groups Overview
| Broker |
Mechanism |
How It Works |
| RabbitMQ |
Competing consumers |
Multiple consumers on same queue; broker distributes round-robin |
| Kafka |
Consumer groups |
Partitions assigned to group members; each partition read by one consumer |
| Redis Streams |
XREADGROUP |
Consumer group tracks last delivered ID per consumer |
Ordering Guarantees
| Broker |
Scope |
Guarantee |
| RabbitMQ |
Per-queue |
Strict FIFO within single queue |
| RabbitMQ |
Across queues |
No ordering guarantee |
| Kafka |
Per-partition |
Strict ordering within partition |
| Kafka |
Across partitions |
No ordering guarantee |
| SQS Standard |
Queue |
Best-effort ordering |
| SQS FIFO |
Message group |
Strict FIFO within group |
| Redis Streams |
Per-stream |
Strict ordering by entry ID |
When to Use Which Broker
| Scenario |
Recommended |
Why |
| Task distribution (email, image processing) |
RabbitMQ |
Flexible routing, competing consumers |
| Event streaming / audit log |
Kafka |
Immutable log, replay, high throughput |
| Simple async in AWS |
SQS |
Managed, no infrastructure |
| Lightweight pub/sub with low latency |
Redis Streams |
Already have Redis, minimal overhead |
| RPC / request-reply |
RabbitMQ |
Built-in reply-to, correlation ID |
| CDC (Change Data Capture) |
Kafka |
Log compaction, connector ecosystem |
| Prioritized processing |
RabbitMQ |
Native priority queues |
| Cross-region replication |
Kafka |
MirrorMaker, built-in replication |
Detection Patterns
# RabbitMQ usage
Grep: "AMQPChannel|PhpAmqpLib|bunny|php-amqplib" --glob "**/*.php"
Grep: "RABBITMQ_|AMQP_" --glob "**/.env*"
# Kafka usage
Grep: "RdKafka|kafka|KafkaConsumer|KafkaProducer" --glob "**/*.php"
Grep: "KAFKA_" --glob "**/.env*"
# SQS usage
Grep: "SqsClient|aws/aws-sdk.*sqs" --glob "**/*.php"
Grep: "SQS_|AWS_SQS" --glob "**/.env*"
# Redis Streams
Grep: "XADD|XREAD|XREADGROUP|XACK" --glob "**/*.php"
Grep: "xAdd|xRead|xReadGroup" --glob "**/*.php"
# Consumer patterns
Grep: "basic_consume|consume\(|poll\(" --glob "**/*.php"
Grep: "basic_ack|basic_nack|commitAsync|xAck" --glob "**/*.php"
# Dead letter configuration
Grep: "dead.letter|x-dead-letter|DLQ|deadLetter" --glob "**/*.php"
References
For detailed information, load these reference files:
references/rabbitmq-advanced.md — Queue types, exchange topologies, clustering, monitoring, PHP patterns
references/kafka-advanced.md — Partitioning, consumer groups, schema registry, exactly-once, PHP patterns
1---2name: message-queue-knowledge3description: Message Queue knowledge base. Provides broker comparison, delivery guarantees, consumer groups, and advanced RabbitMQ/Kafka patterns for messaging audits and generation.4---56# Message Queue Knowledge Base78Quick reference for message broker operations and advanced messaging patterns. Focuses on broker-level operations — for event-driven patterns, see `eda-knowledge`.910## Broker Comparison1112| Feature | RabbitMQ | Apache Kafka | Amazon SQS | Redis Streams |13|---------|----------|-------------|------------|---------------|14| Model | Message queue | Event log | Message queue | Event log |15| Ordering | Per-queue FIFO | Per-partition | Best-effort (FIFO available) | Per-stream |16| Retention | Until consumed | Time/size-based | 4-14 days | Memory/size-based |17| Replay | No (once consumed) | Yes (offset seek) | No | Yes (ID-based) |18| Consumer Groups | Competing consumers | Native support | Not built-in | Native (XREADGROUP) |19| Throughput | ~50K msg/s | ~1M msg/s | ~3K msg/s per queue | ~100K msg/s |20| Latency | Sub-millisecond | Low milliseconds | 10-100ms | Sub-millisecond |21| Protocol | AMQP 0-9-1 | Custom (TCP) | HTTP/SQS API | RESP |22| Clustering | Quorum queues | Built-in (ZK/KRaft) | Managed | Redis Cluster |23| Best for | Task queues, RPC | Event streaming, logs | Serverless, AWS-native | Lightweight streaming |2425## Message Delivery Guarantees2627| Guarantee | Description | Implementation | Trade-off |28|-----------|-------------|----------------|-----------|29| At-most-once | Message may be lost | Fire-and-forget, no ack | Fastest, data loss possible |30| At-least-once | Message delivered 1+ times | Ack after processing | Requires idempotent consumers |31| Exactly-once | Message processed exactly once | Transactional + deduplication | Slowest, most complex |3233### Achieving At-Least-Once in PHP3435```php36// RabbitMQ: manual acknowledgment37$channel->basic_consume(38 queue: 'orders',39 no_ack: false, // require explicit ack40 callback: function (AMQPMessage $msg) use ($channel): void {41 try {42 $this->handler->handle(json_decode($msg->getBody(), true));43 $channel->basic_ack($msg->getDeliveryTag());44 } catch (\Throwable $e) {45 $channel->basic_nack($msg->getDeliveryTag(), requeue: true);46 }47 },48);49```5051## Consumer Groups Overview5253| Broker | Mechanism | How It Works |54|--------|-----------|-------------|55| RabbitMQ | Competing consumers | Multiple consumers on same queue; broker distributes round-robin |56| Kafka | Consumer groups | Partitions assigned to group members; each partition read by one consumer |57| Redis Streams | XREADGROUP | Consumer group tracks last delivered ID per consumer |5859## Ordering Guarantees6061| Broker | Scope | Guarantee |62|--------|-------|-----------|63| RabbitMQ | Per-queue | Strict FIFO within single queue |64| RabbitMQ | Across queues | No ordering guarantee |65| Kafka | Per-partition | Strict ordering within partition |66| Kafka | Across partitions | No ordering guarantee |67| SQS Standard | Queue | Best-effort ordering |68| SQS FIFO | Message group | Strict FIFO within group |69| Redis Streams | Per-stream | Strict ordering by entry ID |7071## When to Use Which Broker7273| Scenario | Recommended | Why |74|----------|-------------|-----|75| Task distribution (email, image processing) | RabbitMQ | Flexible routing, competing consumers |76| Event streaming / audit log | Kafka | Immutable log, replay, high throughput |77| Simple async in AWS | SQS | Managed, no infrastructure |78| Lightweight pub/sub with low latency | Redis Streams | Already have Redis, minimal overhead |79| RPC / request-reply | RabbitMQ | Built-in reply-to, correlation ID |80| CDC (Change Data Capture) | Kafka | Log compaction, connector ecosystem |81| Prioritized processing | RabbitMQ | Native priority queues |82| Cross-region replication | Kafka | MirrorMaker, built-in replication |8384## Detection Patterns8586```bash87# RabbitMQ usage88Grep: "AMQPChannel|PhpAmqpLib|bunny|php-amqplib" --glob "**/*.php"89Grep: "RABBITMQ_|AMQP_" --glob "**/.env*"9091# Kafka usage92Grep: "RdKafka|kafka|KafkaConsumer|KafkaProducer" --glob "**/*.php"93Grep: "KAFKA_" --glob "**/.env*"9495# SQS usage96Grep: "SqsClient|aws/aws-sdk.*sqs" --glob "**/*.php"97Grep: "SQS_|AWS_SQS" --glob "**/.env*"9899# Redis Streams100Grep: "XADD|XREAD|XREADGROUP|XACK" --glob "**/*.php"101Grep: "xAdd|xRead|xReadGroup" --glob "**/*.php"102103# Consumer patterns104Grep: "basic_consume|consume\(|poll\(" --glob "**/*.php"105Grep: "basic_ack|basic_nack|commitAsync|xAck" --glob "**/*.php"106107# Dead letter configuration108Grep: "dead.letter|x-dead-letter|DLQ|deadLetter" --glob "**/*.php"109```110111## References112113For detailed information, load these reference files:114115- `references/rabbitmq-advanced.md` — Queue types, exchange topologies, clustering, monitoring, PHP patterns116- `references/kafka-advanced.md` — Partitioning, consumer groups, schema registry, exactly-once, PHP patterns