Job Reliability
Purpose
Provide a standardized methodology for making background jobs and message consumers reliable under the delivery guarantee that nearly every queue actually offers: at-least-once. That guarantee has two consequences every consumer must handle: duplicates (the same message is delivered more than once) and partial failure (the worker crashes after doing the work but before acknowledging). Code that ignores either will silently double-charge, double-send, lose messages, or stall. This skill catalogs the gaps and the fixes, statically, from the worker source.
The at-least-once contract
- Duplicates are guaranteed, eventually. A network blip, a redelivery after a visibility timeout, or a consumer restart re-delivers a message. → Make consumers idempotent (see
references/dedup-strategies.md).
- Ack means "I'm done," not "I received it." Ack/commit the offset after the side effect succeeds, never before. Ack-then-work loses the message on a crash.
- Poison messages exist. A message that always fails must not retry forever, route it to a dead-letter queue after a bounded number of attempts.
- Retries need backoff + jitter. Immediate infinite retries create a thundering herd against a failing dependency.
The reliability checklist (audit every consumer against this)
| Axis |
The gap |
The fix |
| Idempotency |
Side effect runs again on redelivery (double charge/email) |
Dedup store keyed by message-id/business key; check before acting |
| Ack ordering |
Ack/commit before the work completes |
Ack after success; nack/retry on failure |
| Dead-letter |
Poison message retries forever or is dropped |
DLQ / dead-set after N attempts; alert on it |
| Retry policy |
No backoff, or fixed immediate retry |
Exponential backoff with jitter, capped attempts |
| Visibility/redelivery |
Work outlives the visibility timeout → concurrent redelivery |
Timeout ≥ processing time, or extend/heartbeat |
| Atomic "work + ack" |
Crash between work and ack |
Idempotency makes the inevitable redelivery safe |
| Outbox |
DB commit then publish (dual write), crash loses the event |
Transactional outbox: write event in the same tx; a relay publishes |
Ordering: the crux of correctness
The single most important line to read in any consumer is where the ack/commit happens relative to the side effect:
receive → ack → do work : loses work on a crash after ack. Wrong for anything that matters.receive → do work → ack : at-least-once done right, a crash before ack redelivers, and idempotency absorbs the duplicate.receive → do work → publish → ack, where publish and a DB write must both happen: needs the outbox, or a crash between the two loses the event.
The transactional outbox (when an event must not be lost)
Writing to the database and publishing to a broker are two systems, you cannot commit both atomically. If you COMMIT then publish, a crash in between loses the event; if you publish then commit, a rollback emits a phantom event. The fix: write the event row into an outbox table in the same transaction as the business change; a separate relay (poller or CDC/log-tailing) reads the outbox and publishes, marking rows sent. The publish becomes at-least-once (so consumers still dedup), but the event can never be lost.
Additional Resources
Reference Files
references/queue-patterns.md: per broker (BullMQ, Celery, Sidekiq, RabbitMQ, AWS SQS, Google Pub/Sub, Kafka): the ack model, how to configure retries/backoff, the DLQ mechanism, visibility-timeout/redelivery handling, and the idiomatic idempotent-consumer shape, plus the transactional-outbox recipe.references/dedup-strategies.md: deduplication approaches: dedup stores (Redis SET NX, SQL unique constraint), message-id vs idempotency-key vs business-key dedup, dedup windows, and how to build an exactly-once effect on top of at-least-once delivery.
1---2name: job-reliability3description: This skill should be used when the user mentions "background jobs", "queue", "worker", "dead letter", "DLQ", "retry", "at-least-once", "exactly once", "outbox", "idempotent consumer", "visibility timeout", "BullMQ", "Celery", "Sidekiq", "SQS", "Pub/Sub", "Kafka", or "RabbitMQ", or auditing background-job/message-consumer code for reliability. It provides at-least-once handling patterns, DLQ/backoff/visibility recipes per broker, and message-deduplication strategies.4---56# Job Reliability78## Purpose9Provide a standardized methodology for making background jobs and message consumers reliable under the delivery guarantee that nearly every queue actually offers: **at-least-once**. That guarantee has two consequences every consumer must handle: **duplicates** (the same message is delivered more than once) and **partial failure** (the worker crashes after doing the work but before acknowledging). Code that ignores either will silently double-charge, double-send, lose messages, or stall. This skill catalogs the gaps and the fixes, statically, from the worker source.1011## The at-least-once contract12- **Duplicates are guaranteed, eventually.** A network blip, a redelivery after a visibility timeout, or a consumer restart re-delivers a message. → **Make consumers idempotent** (see `references/dedup-strategies.md`).13- **Ack means "I'm done," not "I received it."** Ack/commit the offset **after** the side effect succeeds, never before. Ack-then-work loses the message on a crash.14- **Poison messages exist.** A message that always fails must not retry forever, route it to a **dead-letter queue** after a bounded number of attempts.15- **Retries need backoff + jitter.** Immediate infinite retries create a thundering herd against a failing dependency.1617## The reliability checklist (audit every consumer against this)1819| Axis | The gap | The fix |20|---|---|---|21| Idempotency | Side effect runs again on redelivery (double charge/email) | Dedup store keyed by message-id/business key; check before acting |22| Ack ordering | Ack/commit **before** the work completes | Ack **after** success; nack/retry on failure |23| Dead-letter | Poison message retries forever or is dropped | DLQ / dead-set after N attempts; alert on it |24| Retry policy | No backoff, or fixed immediate retry | Exponential backoff **with jitter**, capped attempts |25| Visibility/redelivery | Work outlives the visibility timeout → concurrent redelivery | Timeout ≥ processing time, or extend/heartbeat |26| Atomic "work + ack" | Crash between work and ack | Idempotency makes the inevitable redelivery safe |27| Outbox | DB commit then publish (dual write), crash loses the event | Transactional outbox: write event in the same tx; a relay publishes |2829## Ordering: the crux of correctness30The single most important line to read in any consumer is **where the ack/commit happens relative to the side effect**:31- `receive → ack → do work` : **loses** work on a crash after ack. Wrong for anything that matters.32- `receive → do work → ack` : at-least-once done right, a crash before ack redelivers, and idempotency absorbs the duplicate.33- `receive → do work → publish → ack`, where publish and a DB write must both happen: needs the **outbox**, or a crash between the two loses the event.3435## The transactional outbox (when an event must not be lost)36Writing to the database **and** publishing to a broker are two systems, you cannot commit both atomically. If you `COMMIT` then publish, a crash in between loses the event; if you publish then commit, a rollback emits a phantom event. The fix: write the event row into an `outbox` table **in the same transaction** as the business change; a separate relay (poller or CDC/log-tailing) reads the outbox and publishes, marking rows sent. The publish becomes at-least-once (so consumers still dedup), but the event can never be lost.3738## Additional Resources39### Reference Files40- **`references/queue-patterns.md`**: per broker (BullMQ, Celery, Sidekiq, RabbitMQ, AWS SQS, Google Pub/Sub, Kafka): the ack model, how to configure retries/backoff, the DLQ mechanism, visibility-timeout/redelivery handling, and the idiomatic idempotent-consumer shape, plus the transactional-outbox recipe.41- **`references/dedup-strategies.md`**: deduplication approaches: dedup stores (Redis `SET NX`, SQL unique constraint), message-id vs idempotency-key vs business-key dedup, dedup windows, and how to build an exactly-once **effect** on top of at-least-once delivery.
sigistry