# Background Jobs

> Run work off the request thread reliably — queue design, retries with exponential backoff + jitter, dead-letter queues, concurrency control, and idempotent handlers. Use when an operation is slow/external, when jobs fail silently, or when retries cause duplicates. Not for write+event transactional reliability — the dual-write problem (use async-messaging Outbox) or webhook-receiver specifics (use webhook-design).

- Skill: `jaykim88/background-jobs` (Agent Skill)
- Install (CLI): `npx skillmds@latest add jaykim88/background-jobs`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jaykim88/background-jobs/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- License: MIT
- Author: JayKim88 (https://skillmd.com/u/jaykim88)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/jaykim88/background-jobs

---


# Background Jobs

## Purpose
Move slow, external, or deferrable work off the request path into a durable queue with proper retries, dead-lettering, and idempotent handlers — so failures are visible and recoverable, not silent.

**Universal** — queue design, retry/backoff, DLQ, concurrency control, and idempotent handlers are job-processing principles; BullMQ is the default implementation.

## Procedure

1. **Decide what belongs in a job**
   - Slow (> ~1s), external I/O, deferrable, or retryable work → job
   - Keep the request fast; return early, do the work async
   - Don't queue work that must be synchronous for the user's next action

2. **Make every handler idempotent**
   - At-least-once execution means a job CAN run more than once (retry, redelivery)
   - Dedupe on a job/business key; design the side effect to be safe to repeat (see `resilience-patterns` idempotency)

3. **Configure retries: exponential backoff + jitter, capped**
   - Set `attempts` + `backoff: { type: 'exponential' }` (+ jitter)
   - Distinguish retryable (timeout, 503) from non-retryable (validation, 4xx) failures — don't retry the un-retryable

4. **Dead-letter queue for poison messages**
   - After max attempts, move to a DLQ instead of dropping or infinite-retrying
   - Alert on DLQ growth; provide a replay path after the bug is fixed

5. **Control concurrency — and apply backpressure**
   - Set worker concurrency to protect downstream (DB pool, rate-limited API)
   - Rate-limit jobs hitting a quota'd external service
   - **Producers outpacing consumers** → queue depth climbs. Decide the policy: producer slowdown / circuit-break the producing endpoint / shed (return 429 / drop low-priority). Unbounded queue depth is a delayed outage, not a working system

5b. **Set a visibility timeout / heartbeat for long jobs**
   - A worker that dies mid-job leaves the job in "in progress" — without a visibility timeout it's stuck forever; with one (and an active heartbeat from healthy workers) the job becomes eligible for re-delivery
   - Tune the timeout to slightly above the job's p99 duration; emit heartbeats for genuinely-long jobs

5c. **Retain only what you need (job table cleanup)**
   - Completed and failed jobs accumulate — set `removeOnComplete` (or equivalent) with a retention count/age; archive critical job history elsewhere if needed
   - An unbounded job store eats Redis / DB memory silently

6. **Make jobs observable**
   - Log job start/end/failure with correlation id (see `observability-setup`)
   - Track queue depth, processing latency, failure rate

7. **Validate (validation loop)**
   - Force a job to fail → verify retry with backoff, then DLQ after cap (not infinite, not silent)
   - Run a job twice → verify idempotent (no duplicate side effect)
   - If a duplicate side effect occurs → handler not idempotent; fix and re-test

## Anti-patterns

| ❌ Anti-pattern | ✅ Correct |
|---|---|
| Non-idempotent handler (retry double-charges) | Idempotent handler (dedupe key) |
| Retry forever on any failure | Capped attempts → DLQ; retry only transient errors |
| Failed jobs silently dropped | DLQ + alert + replay path |
| Unbounded concurrency exhausting DB pool | Tuned concurrency + rate limits |
| Blocking the request on slow work | Enqueue, return fast |
| No visibility timeout / heartbeat → job stuck "in progress" forever on worker crash | Visibility timeout slightly above p99; heartbeat for long jobs |
| Completed jobs accumulating in Redis until OOM | `removeOnComplete` retention + archive critical history elsewhere |
| Producer faster than consumers; queue grows unbounded | Backpressure policy (slow producer / 429 / shed low-priority) |

## Severity tiers

| Tier | Examples | Action SLA |
|---|---|---|
| **Critical** | Non-idempotent job double-charging on retry; failed payment jobs silently dropped | Fix immediately |
| **Major** | No DLQ (infinite retry or data loss); no backoff (retry storm) | Fix this sprint |
| **Minor** | Concurrency untuned; missing queue-depth metric | Schedule within 2 sprints |

## Completion Criteria
- [ ] Handlers idempotent (verified by double-run)
- [ ] Retries capped with exponential backoff + jitter
- [ ] DLQ configured + alerted
- [ ] Concurrency tuned to protect downstream
- [ ] Queue depth + failure rate observable

## Output
- **Job definitions + worker config**: retry/backoff/concurrency/DLQ
- **Idempotency keys** per job type
- **Commit format**: `feat(jobs): add <job> with DLQ + backoff` / `fix(jobs): make <handler> idempotent`

## Implementation

### TypeScript + BullMQ + Redis (default)
- `defaultJobOptions: { attempts: 5, backoff: { type: 'exponential', delay: 1000 } }` (+ jitter via custom strategy)
- DLQ: a separate queue; move on `failed` after attempts exhausted; BullMQ `failed` events
- Concurrency: `new Worker(name, fn, { concurrency: N, limiter: { max, duration } })`
- Idempotent: dedupe table on job key, or use `jobId` for natural dedup
- See [BullMQ "Going to Production"](https://docs.bullmq.io/guide/going-to-production)

### Other stacks
- **Python**: Celery (`acks_late=True`, `max_retries`, `autoretry_for`) + dead-letter via routing
- **Go**: Asynq or River — same retry/DLQ/concurrency model
- **Universal**: idempotency + capped backoff + DLQ are queue-agnostic; the at-least-once contract is the same everywhere

## Related skills
- `async-messaging` — the outbox relay and event consumers run as jobs
- `resilience-patterns` — job retries reuse backoff + idempotency
- `webhook-design` — received webhooks are processed as jobs

## Reference
- **Key insight encoded**: Configure exponential backoff with jitter in `defaultJobOptions`, cap attempts, and make handlers idempotent — at-least-once execution means a job can run more than once on retry.