# Background Job Patterns

> Use when reviewing, designing, or debugging background job / async queue processing in any language or stack — Sidekiq, BullMQ, Asynq, River, Faktory, JobRunr, Quartz, Laravel Queue, Symfony Messenger, or a custom queue. Trigger on symptoms like duplicate job execution, jobs stuck active/busy, retry storms, worker memory growth, jobs missing after deploy, or connection pool exhaustion from workers.

- Skill: `gabrielras/background-job-patterns` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add gabrielras/background-job-patterns`
- Raw SKILL.md: https://api.skillmd.com/api/skills/gabrielras/background-job-patterns/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Marketing & Growth
- Author: gabrielras (https://skillmd.com/u/gabrielras)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/gabrielras/background-job-patterns

---


# Background Job & Queue Patterns

## Overview

Every queue system — regardless of language or backend (Redis, Postgres, a message broker) — solves the same core problem: **deliver work at least once to a worker that might crash, hang, or run twice.** The same 14 patterns fix this in any stack; only the syntax changes.

**Core principle:** the pattern is the reusable unit, not any one library's implementation.

## When to Use

- Reviewing code that enqueues or processes jobs, in any language
- Diagnosing: duplicate side effect on retry, job stuck "active"/"busy" forever, retries landing in a synchronized burst against an external API, worker memory that only climbs, a job silently missing after a deploy, DB connection pool exhausted by workers
- Choosing a queue architecture (Redis vs Postgres `SKIP LOCKED` vs a real message broker)

## Quick Reference

| Pattern | Prevents | Detail |
|---|---|---|
| Idempotency key | Duplicate side effects on retry | patterns.md #1 |
| Visibility timeout / lease | Job "lost" if worker dies mid-run | patterns.md #2 |
| Dead-letter queue | Infinite retry loop | patterns.md #3 |
| Backoff + jitter | Retry storm / thundering herd | patterns.md #4 |
| Concurrency ≤ weakest downstream resource | Connection exhaustion | patterns.md #5 |
| Transactional outbox | Job fired before commit, or lost after a crash | patterns.md #7 |
| Competing consumers | Throughput capped by a single worker | patterns.md #13 |
| Circuit breaker | Retrying against a dependency that's already down | patterns.md #14 |

## Diagnosis Flow

A symptom almost always maps to one broken pattern. Check in this order:

1. **Duplicate side effect?** → missing idempotency key (pattern 1)
2. **Job stuck "active"/"busy" forever?** → worker died without releasing its lock — check lease/lock duration (pattern 2; the mechanism's name differs per stack, but it's always the same reservation-with-expiry)
3. **Retries landing in a synchronized burst?** → backoff without jitter (pattern 4)
4. **Worker memory only climbs?** → before assuming a leak, confirm: allocator fragmentation, an unreleased C-level handle in a long-running process, or a genuine leak — the root cause differs per runtime, the symptom doesn't
5. **Connection pool exhausted, but queue throughput looks fine?** → concurrency not bounded by the weakest downstream resource (pattern 5)
6. **Jobs silently missing after a deploy?** → missing explicit restart signal for the worker (pattern 10)

## Reference

- **references/patterns.md** — all 14 patterns in full: core principle, when to use, ❌ BEFORE / ✅ AFTER pseudocode, common mistakes

## Common Mistakes

- Assuming the queue guarantees order — most don't by default (pattern 8)
- Treating any unhandled failure as "just that job failed" without confirming the runtime's concurrency unit — in some models it kills the entire process (pattern 6)
- Recommending a queue library without checking it's still actively maintained