Queues
Purpose
Choose and design the queue transport that background-jobs runs on: broker, queue topology, worker model, delivery guarantees, dead-letter handling, and operational visibility.
When to Use
- When background jobs need real infrastructure (beyond a single in-process timer).
- Not for defining job idempotency/retry behavior (
background-jobs) or time triggers (scheduled-jobs).
Inputs
- Job inventory (
background-jobs): volume, latency tolerance, ordering needs, must-happen classification.
- Deployment shape and existing infrastructure (Redis present? cloud provider?).
Discovery Questions
- Throughput and burst profile per job type? Latency tolerance (seconds vs hours)?
- Ordering requirements (per-entity ordering vs none)?
- What infrastructure already exists — is Redis/BullMQ, a cloud queue (SQS), or a broker (RabbitMQ) the low-operational-cost fit?
- Who watches the queues (dashboards, alerts)?
Responsibilities
- Choose the broker on requirements + existing ops surface: BullMQ/Redis (Node-native, common default when Redis exists), managed cloud queues (low ops, at-least-once), RabbitMQ/Kafka-class only for routing/streaming needs that justify them — recorded with trade-offs (Gate 2 if it adds infrastructure).
- Design topology: queues per job class (not one global queue) so priorities/concurrency/rates tune independently; name conventions.
- Design workers: separate worker processes from the API (independent scaling/deploys), concurrency per queue, graceful shutdown (finish/return in-flight jobs on deploy).
- Set delivery semantics expectations: assume at-least-once; visibility timeouts/lock durations sized to job runtime; stalled-job recovery.
- Configure dead-letter queues per queue with alerting and a replay procedure (
backend-observability).
- Plan backpressure: max queue depth alarms, producer behavior when saturated.
Required Workflow
- Derive throughput/ordering/latency needs from the job inventory.
- Choose broker with trade-offs; record for approval if new infrastructure.
- Define queue topology + per-queue concurrency/priority/rate.
- Define worker deployment, shutdown, and stall recovery.
- Configure DLQs, alerts, dashboards, replay runbook.
- Specify tests: job survives worker crash, stalled job recovers, DLQ receives terminal failures.
Decision Rules
- Don't add a broker class the team can't operate; managed/existing beats theoretically-better.
- One queue per job class; mixing critical and bulk work in one queue starves the critical.
- Visibility/lock timeout > worst-case job runtime, or jobs double-run mid-flight (
background-jobs idempotency still required).
- Per-entity ordering needs are real design constraints (keyed queues/serialization) — don't fake with sleeps.
Rules
- Queue infrastructure is a stack decision — user approval before adding it (
../../stack-recommendation).
- Workers deploy and scale independently of the API.
- Every queue has a DLQ + alert; silent terminal loss is a defect.
Anti-Patterns
- One global queue for everything.
- Workers inside the API process, dying with each deploy mid-job.
- Assuming exactly-once delivery.
- No DLQ — terminal failures retried forever or dropped.
- Unbounded queue growth with no alarms.
Validation Checklist
Definition of Done
A recorded queue design — broker with trade-offs, per-class topology, worker lifecycle, at-least-once handling, DLQs with alerting and replay — ready to carry the job inventory.
Related Skills
background-jobs, scheduled-jobs, backend-observability, backend-deployment (worker deploys), ../../stack-recommendation, nestjs-foundation (Nest queue module when applicable).
Related Knowledge
../../../knowledge/ (existing infrastructure, ops capacity).
Related References
../../../references/backend/jobs/ (topology notes, when populated).
Context Loading Guidance
- Requires: job inventory with volumes/ordering, infrastructure summary.
- Does not require: individual job logic, unrelated endpoints.
- May load:
background-jobs (semantics alignment), backend-observability.
- Stop when: broker, topology, worker, and DLQ design are recorded.
Token Efficiency Guidance
Decide from the job-inventory table; per-queue settings as a table. Link broker docs, don't restate them.
1---2name: queues3description: Use to select and design queue infrastructure — broker choice (e.g. BullMQ/Redis, SQS, RabbitMQ), queue topology, workers/concurrency, delivery semantics, dead-letter queues, and monitoring. Job behavior itself is the background-jobs skill.4---56# Queues78## Purpose910Choose and design the queue transport that `background-jobs` runs on: broker, queue topology, worker model, delivery guarantees, dead-letter handling, and operational visibility.1112## When to Use1314- When background jobs need real infrastructure (beyond a single in-process timer).15- **Not** for defining job idempotency/retry behavior (`background-jobs`) or time triggers (`scheduled-jobs`).1617## Inputs1819- Job inventory (`background-jobs`): volume, latency tolerance, ordering needs, must-happen classification.20- Deployment shape and existing infrastructure (Redis present? cloud provider?).2122## Discovery Questions2324- Throughput and burst profile per job type? Latency tolerance (seconds vs hours)?25- Ordering requirements (per-entity ordering vs none)?26- What infrastructure already exists — is Redis/BullMQ, a cloud queue (SQS), or a broker (RabbitMQ) the low-operational-cost fit?27- Who watches the queues (dashboards, alerts)?2829## Responsibilities3031- Choose the broker on requirements + existing ops surface: BullMQ/Redis (Node-native, common default when Redis exists), managed cloud queues (low ops, at-least-once), RabbitMQ/Kafka-class only for routing/streaming needs that justify them — recorded with trade-offs (Gate 2 if it adds infrastructure).32- Design topology: queues per job class (not one global queue) so priorities/concurrency/rates tune independently; name conventions.33- Design workers: separate worker processes from the API (independent scaling/deploys), concurrency per queue, graceful shutdown (finish/return in-flight jobs on deploy).34- Set delivery semantics expectations: assume **at-least-once**; visibility timeouts/lock durations sized to job runtime; stalled-job recovery.35- Configure **dead-letter queues** per queue with alerting and a replay procedure (`backend-observability`).36- Plan backpressure: max queue depth alarms, producer behavior when saturated.3738## Required Workflow39401. Derive throughput/ordering/latency needs from the job inventory.412. Choose broker with trade-offs; record for approval if new infrastructure.423. Define queue topology + per-queue concurrency/priority/rate.434. Define worker deployment, shutdown, and stall recovery.445. Configure DLQs, alerts, dashboards, replay runbook.456. Specify tests: job survives worker crash, stalled job recovers, DLQ receives terminal failures.4647## Decision Rules4849- Don't add a broker class the team can't operate; managed/existing beats theoretically-better.50- One queue per job class; mixing critical and bulk work in one queue starves the critical.51- Visibility/lock timeout > worst-case job runtime, or jobs double-run mid-flight (`background-jobs` idempotency still required).52- Per-entity ordering needs are real design constraints (keyed queues/serialization) — don't fake with sleeps.5354## Rules5556- Queue infrastructure is a stack decision — user approval before adding it (`../../stack-recommendation`).57- Workers deploy and scale independently of the API.58- Every queue has a DLQ + alert; silent terminal loss is a defect.5960## Anti-Patterns6162- One global queue for everything.63- Workers inside the API process, dying with each deploy mid-job.64- Assuming exactly-once delivery.65- No DLQ — terminal failures retried forever or dropped.66- Unbounded queue growth with no alarms.6768## Validation Checklist6970- [ ] Broker chosen on requirements/ops fit; recorded (+ approval if new infra).71- [ ] Topology: queue per class, concurrency/priority per queue.72- [ ] Worker lifecycle: separate processes, graceful shutdown, stall recovery.73- [ ] At-least-once semantics acknowledged; timeouts sized.74- [ ] DLQs + alerts + replay runbook.75- [ ] Crash/stall/DLQ tests specified.7677## Definition of Done7879A recorded queue design — broker with trade-offs, per-class topology, worker lifecycle, at-least-once handling, DLQs with alerting and replay — ready to carry the job inventory.8081## Related Skills8283`background-jobs`, `scheduled-jobs`, `backend-observability`, `backend-deployment` (worker deploys), `../../stack-recommendation`, `nestjs-foundation` (Nest queue module when applicable).8485## Related Knowledge8687`../../../knowledge/` (existing infrastructure, ops capacity).8889## Related References9091`../../../references/backend/jobs/` (topology notes, when populated).9293## Context Loading Guidance9495- **Requires:** job inventory with volumes/ordering, infrastructure summary.96- **Does not require:** individual job logic, unrelated endpoints.97- **May load:** `background-jobs` (semantics alignment), `backend-observability`.98- **Stop when:** broker, topology, worker, and DLQ design are recorded.99100## Token Efficiency Guidance101102Decide from the job-inventory table; per-queue settings as a table. Link broker docs, don't restate them.
ahtishamshahzad