Task Queues And Competing Consumers

Task Queues And Competing Consumers

Purpose

Decide whether work belongs on a queue consumed by a pool of interchangeable workers, and then operate that pool so the queue's own mechanics do not corrupt the work. The queue owns assignment through pull or broker delivery/credit. Ready workers can take more work, but prefetch, task size, priority and dispatch policy determine actual balance.

The failure this prevents is silent double-execution. In an SQS-style visibility model, receiving does not remove a message; it hides it for a bounded time, and when that time expires it becomes visible again for another worker. If the first worker is still running — slow dependency, long GC pause, a batch that grew — the message is now being processed twice, concurrently, with nothing failing and nothing retrying. The visibility timeout is a bet on how long the work takes, and losing the bet duplicates the work.

Workflow

Inspect broker/queue type, acknowledgement mode, client/framework and Java versions, prefetch, retention and redelivery configuration. RabbitMQ channel acknowledgements and JMS sessions are not SQS receipt leases. Preserve the deployed baseline. Missing evidence is unknown; deliver the ownership/ack/recovery path, budgets and actual validation. Run crash/requeue experiments only in isolated or already authorized environments.

1. Run the decision block below: interchangeable workers and independent items, or explicit per-key lanes with ordering/recovery semantics.
2. Measure lease exposure, not just handler time: prefetch/permit wait + queue client work + handler + acknowledgement, under degraded dependencies and pauses. Select an explicit premature-redelivery versus crash-recovery objective; there is no universal percentile.
3. Pick one of the three responses to lease expiry and write down which: size from the tail, extend by heartbeat while working, or make the handler repeat-safe (idempotency) and accept the overlap. Most systems need the third regardless.
4. Bound accepted backlog by age, bytes/items, retention and recovery capacity. If the managed broker cannot reject at a depth, enforce admission upstream and specify what the producer sees (rate-limiting-and-load-shedding).
5. Bound intake before delivery: reserve permits before pulling, or configure broker credit/ prefetch and bounded dispatch for push consumers. Handle receive failure and submission rejection without leaking permits or deliveries. The limit is concurrency-limiting-and-bulkheads.
6. Scale from a signal set. Age is closest to a latency SLO, but combine it with depth, arrival/drain rate, in-flight saturation and startup delay; broker age can be approximate or reset by redelivery. references/worker-loop-and-scaling.md gives the control model.
7. Prove it by killing a worker mid-lease and asserting redelivery, one observable side effect under the declared contract, and no unexplained missing item. Happy paths alone do not exercise recovery; sample fault cases do not prove every failure mode.

Decision block

Use a task queue with competing consumers when:
- items commute, or ordering is explicitly enforced by a broker group/partition and the
  worker preserves that lane's ownership
- the worker is stateless and any worker can take any item
- producer and consumer rates differ over time and a bounded buffer absorbs the difference
- the work is retryable and its side effect can be made repeat-safe

Avoid a task queue when:
- correctness needs an order the queue cannot express or preserve through retry/redelivery;
  FIFO/message-group queues can serialize a key, but head-of-line blocking is the cost
- the same item must be consumed independently by several subscribers with their own
  positions, or must be replayable after it succeeded; here an acked message is gone

Prefer a partitioned log instead when:
- per-key ordering, replay, or several independent consumer groups are required
  (kafka-consumers-in-java)

Prefer fenced ownership or resource-side concurrency control when:
- stale concurrent execution would violate correctness. Neither a queue lease nor leader
  election alone proves exactly one effect (leader-election)

Prefer an in-process executor instead when:
- the work need not survive the process (executors-and-task-lifecycle)

Rules

• A visibility timeout is not a lock. It bounds how long a message stays hidden; it excludes nobody, and two workers holding one item is the model working as designed. Mutual exclusion needs a fencing token the resource checks (leader-election) — a lease alone does not survive a GC pause on its holder.
• Size from the measured receive-to-ack distribution plus safety/resolution margin, against a stated premature-redelivery error budget and maximum crash-recovery delay. Segment by task class; censored timings from already-expired work do not reveal the unseen tail.
• A heartbeat that extends the lease can keep wedged work hidden until renewal stops or a broker limit is reached. Cap total lease time and use credible progress where available, never thread liveness as proof of progress.
• A batch fetch starts every lease at receive time. For B records processed serially, the last sees the sum of preceding durations; with C handler slots it waits behind roughly ceil(B/C)-1 waves, but correlated tails and scheduling matter. Measure receive-to-start and receive-to-ack, reduce prefetch, or extend per message—do not multiply one percentile and call it a probabilistic bound.
• nack with immediate requeue and no delay is a hot loop: the same message returns instantly, fails again, and the pool spends its capacity on one item. Requeue with a delay and a delivery counter, and route it to poison-messages-and-dlq at the threshold.
• Never write while (true) { var msg = poll(); executor.submit(() -> handle(msg)); } onto an unbounded executor. It drains the broker's queue into the heap: the queue's backpressure disappears, depth reads zero while the process is overloaded, and every in-flight lease is on the clock at once. Acquire the permit before poll.
• Queue depth alone cannot predict wait, while oldest-message-age alone can be stale, approximate, reset by retry, or dominated by one poison item. Use age for SLO alerting and a controller signal set—visible/in-flight depth, arrival/drain rate, service-time distribution, saturation, startup delay and downstream capacity. Validate stability and scale-down hysteresis.
• A shared queue can balance work dynamically, but it is not the per-worker-deque work-stealing algorithm. Prefetch can strand work behind slow handlers; measure distribution and credit. The in-JVM mechanics are forkjoinpool-and-work-stealing.
• Strict priority starves the low class permanently while high-priority arrivals sustain above capacity. Bound the starvation explicitly — age items into a higher class after a stated time in queue, or give each class a weighted share of workers. "Rarely happens" is not a policy.
• On shutdown, stop intake and resolve polls racing with shutdown, then drain or cancel held work. Releasing a delivery while its old handler still runs invites overlap; retain resource guards/idempotency. Interruption does not prove termination. The grace budget and ordering are kubernetes-service-lifecycle.
• A durable broker may intentionally have no hard depth rejection, but accepted backlog is never economically unbounded. Set maximum useful age, retention/storage quotas and catch-up/recovery objectives; shed or defer admission before work becomes guaranteed-expired.

Security and tenant isolation

• Authenticate producers/workers and authorize queue, task type and tenant; never trust a priority, callback URL, class name or serialized payload merely because it came from a queue.
• Validate size/schema before leasing expensive capacity. Encrypt sensitive payloads, minimize DLQ copies and define deletion/retention for primary, retry and dead-letter queues.
• Apply per-tenant concurrency/quotas so one tenant cannot consume every worker or age another tenant past its deadline. Preserve trace, task, attempt and idempotency identifiers without putting secrets or raw PII in metric labels.

Primary references

• Amazon SQS visibility timeout — redelivery, in-flight limits, FIFO groups and extension limits.
• RabbitMQ consumer acknowledgements — delivery acknowledgement and requeue semantics, which are not identical to SQS visibility.
• JMS acknowledgement modes — session and acknowledgement semantics.

References

• The lease model — choosing the visibility timeout from the processing-time distribution, the duplicate-work window drawn as a sequence, heartbeat extension with its failure mode and its cap, and what to do instead of treating a lease as a lock. Read when setting or reviewing a visibility timeout, or when duplicate side effects appear with no retry in the code.
• Worker loop and scaling — a competing-consumer ownership protocol with bounded concurrency, lease heartbeat and drain-on-shutdown; the autoscaling signal against the wrong ones; priority with ageing; and a test that kills a worker mid-lease. Read before writing or reviewing a worker, or when deciding what the pool scales on.