Background Jobs

Background Jobs

Concept of the skill

A background job system has five primitives: a producer records durable work, a queue orders and deduplicates it, a worker leases and executes it, a state store records progress and outcomes, and a notification path tells humans or systems what changed.

Coverage

• Inline-vs-background execution decisions for web requests, API routes, workers, and serverless functions.
• Durable job contracts: job identity, payload, state, priority, ownership, attempts, progress, result, and failure record.
• Queue and worker patterns: push queues, pull queues, leases, deduplication, priority, rate control, and concurrency limits.
• Reliability patterns: idempotency keys, retry classification, exponential backoff with jitter, checkpoints, dead-letter handling, and partial failure recovery.
• User-visible progress: stage names, percentages, timestamps, cancellation, completion notification, and stale status handling.
• Verification: proving long work left the request path without losing observability or recovery paths.

Philosophy of the skill

Background jobs are not just "run this later." They are a reliability boundary between interactive work and processing work. A request handler is optimized for short, synchronous feedback. A worker is optimized for durable execution, retries, checkpoints, and controlled resource use.

The most common failure is moving code into a worker while keeping request-handler assumptions: no durable state, no idempotency, no progress, no cancellation, and no evidence of completion. That makes the system feel faster only until the first timeout, duplicate enqueue, or partial failure. A good job design makes the execution contract visible before choosing a queue product.

Execution Decision Gate

Use this gate before adding a queue. If any answer in the right column is true, design a background job instead of inline request work.

Rule of thumb: if you need progress, checkpointing, retry classification, cancellation, concurrency control, or delayed completion, you are already designing a background job.

Job Contract

Every background job needs a durable contract. The exact storage can be a database row, queue message plus result table, workflow engine state, or object store record, but the same fields need clear ownership.

State Machine

Keep states few and explicit:

queued -> running -> succeeded
                  -> retry_waiting -> running
                  -> failed
                  -> cancelled

Use failed for terminal failure after retry policy is exhausted. Use cancelled only when the system intentionally stops work. Do not collapse retryable and terminal failures into one ambiguous error state.

Queue And Worker Patterns

Lease-Based Pull Worker

Use a lease when workers pull from a shared store. The lease prevents two workers from processing the same job while still letting another worker recover abandoned work after the lease expires.

async function claimNextJob(workerId: string) {
  return updateOneJob(
    {
      status: "queued",
      runAfter: { lte: new Date() },
    },
    {
      status: "running",
      leaseOwner: workerId,
      leaseExpiresAt: new Date(Date.now() + 5 * 60 * 1000),
      startedAt: new Date(),
    },
    { sort: { priority: 1, createdAt: 1 } },
  );
}

Reliability Patterns

Idempotency

Background workers must assume at-least-once execution. A retry, duplicate enqueue, worker crash, or lease expiry can run the same logical job more than once.

Use an idempotency key that represents the logical work, not the physical attempt:

const idempotencyKey = `report:${workspaceId}:${periodStart}:${periodEnd}`;

await enqueueJob({
  type: "report.generate",
  idempotencyKey,
  payload: { workspaceId, periodStart, periodEnd },
});

The worker should also make side effects idempotent. Deduplicating enqueue is helpful but not sufficient because messages can be delivered more than once.

Retry Classification

Not every failure deserves a retry.

Use jitter so a shared outage does not cause every worker to retry at the same moment:

function retryDelayMs(attempt: number) {
  const base = 1000;
  const cap = 5 * 60 * 1000;
  const exponential = Math.min(base * 2 ** attempt, cap);
  const jitter = Math.floor(Math.random() * base);
  return exponential + jitter;
}

Checkpointing

Long jobs need resumable checkpoints. A checkpoint should identify the last committed unit of work, not just a percentage.

async function processPages(jobId: string) {
  let cursor = await loadCheckpoint(jobId);

  while (true) {
    const page = await fetchNextPage(cursor);
    if (page.items.length === 0) break;

    await processBatch(page.items);
    cursor = page.nextCursor;
    await saveCheckpoint(jobId, cursor);
    await updateProgress(jobId, { stage: "processing", processed: page.totalProcessed });
  }
}

Progress Throttling

Progress writes are product value only when they communicate meaningful change. Updating progress after every item in a large batch can overload the same database or cache the job is trying to use.

Use one of these gates:

• Update every N items.
• Update every T seconds.
• Update when the stage changes.
• Update at completion or terminal failure.

User-Facing Progress

The UI does not need internal worker details. It needs a stable status contract.

Avoid fake precision. If you do not know the denominator, report stages or processed counts instead of a misleading percentage.

Concurrency And Priority

Concurrency limits protect shared resources. Define at least one limit before shipping a worker:

Priority should reorder queued work, not bypass safety. A high-priority job still needs idempotency, leases, and retry policy.

Observability

Background jobs need enough telemetry to answer four questions without reading code:

• Was the job enqueued?
• Did a worker claim it?
• What progress or checkpoint was last committed?
• Did it succeed, fail terminally, retry, or get cancelled?

Log job ID, type, attempt number, state transitions, duration, failure class, and queue latency. Emit metrics for queue depth, age of oldest queued job, success rate, retry rate, terminal failure rate, and worker saturation. Trace multi-step jobs when a single user action fans out into several worker operations.

Verification

After applying this skill, verify:

• Long or unbounded work is outside the interactive request path.
• Every enqueued job has a durable status that can be queried after refresh or worker restart.
• The job contract includes idempotency, attempts, progress, result, and failure reason.
• Retry policy distinguishes transient, rate-limit, validation, permission, partial-progress, and unknown failures.
• Backoff includes jitter or an equivalent herd-prevention mechanism.
• Long jobs checkpoint the last committed unit of work.
• Progress updates are throttled by item count, time, stage, or completion.
• Worker concurrency is bounded globally and at any needed fairness boundary.
• Terminal failure and cancellation are visible to users or operators.
• Tests or manual probes cover duplicate enqueue, retry, resume, and terminal failure behavior.

Do NOT Use When

Anti-Patterns