Backend Background Jobs

Backend Background Jobs

Purpose

Design reliable background job processing with queue topology, retry, concurrency, and monitoring. Every job must be idempotent, retryable, observable, and gracefully handled on failure.

Agent Protocol

Trigger

Exact user phrases: "background job", "task queue", "worker", "job processing", "cron job", "scheduled task", "async task", "job retry", "queue worker", "Sidekiq", "Celery", "Bull", "delayed job", "job priority", "job scheduling".

Input Context

• Job types (fire-and-forget, delayed, scheduled, chained).
• Queue backend (Redis, SQS, database table, RabbitMQ).
• Job duration (affects concurrency and timeout).
• Retry requirements (max retries, retry interval, irrecoverable failure definition).

Output Artifact

Job and worker design as formatted text.

Response Format

// Job contract (interface/type)
// Worker implementation outline

# Queue topology config
# Retry policy
# Concurrency rules

No preamble. No postamble. No explanations. No filler/hedging/transitions. Compress output.

Completion Criteria

• Job defined with idempotency key and full payload schema
• Retry strategy with exponential backoff and jitter configured
• Queue topology defined (default, priority, dead letter, scheduling queues)
• Worker concurrency and graceful shutdown configured
• Monitoring hooks for success/failure/metrics
• Scheduled/cron jobs defined with timezone and error notification

Architecture Decision Trees

Queue Backend Selection

What infrastructure do you already have?
├── Redis already in stack?
│   ├── Jobs < 100KB, sub-minute duration?
│   │   ├── Yes → Bull/BullMQ (built-in priority, delay, DLQ)
│   │   └── No → Consider RabbitMQ for complex routing
│   └── Need persistence and high throughput?
│       └── Bull with Redis Cluster or SQS
├── AWS-native?
│   ├── Jobs fit in 256KB? → SQS (fully managed, unlimited throughput)
│   ├── Need FIFO ordering? → SQS FIFO (exactly-once, 300 TPS)
│   └── Need long-running jobs? → SQS + Step Functions
├── Need transactional consistency with DB?
│   └── Database-backed queue (simplest setup, no infra)
└── Need complex routing (topic exchanges, headers)?
    └── RabbitMQ

Job Type Decision Tree

Does the job need a response or callback?
├── Yes → Is it a multi-step workflow?
│   ├── Yes → Chained jobs with compensation/rollback
│   └── No → Fire-and-forget with monitoring
└── No → Should it run at a specific time?
    ├── Yes → Delayed job (scheduled execution)
    └── No → Should it run on a fixed schedule?
        ├── Yes → Scheduled (cron) job
        └── No → Fire-and-forget job

Retry Strategy Decision Tree

What happens when the job fails?
├── Is the error transient (timeout, network, 503)?
│   ├── Yes → Retry with exponential backoff (5-10 attempts)
│   └── No → Is the error a dependent service failure?
│       ├── Yes → Retry with longer backoff (10-25 attempts)
│       └── No → Non-recoverable → DLQ immediately
├── Does the job have side effects?
│   ├── Yes → Ensure idempotency before retrying
│   └── No → Safe to retry as-is
└── Is the job part of a chain?
    └── Rollback/compensate on failure, retry individual steps

Workflow

Step 1: Job Classification

Fire-and-forget: no callback, best-effort delivery, highest throughput. Delayed: execute after N seconds/minutes using Redis sorted sets or SQS delay queues. Scheduled: cron expression, timezone-aware, prevents overlapping execution. Recurring: fixed interval, always runs regardless of previous success/failure. Chained: next job enqueued after previous completes, with compensation/rollback on failure.

Step 2: Queue Backend Selection

Step 3: Queue Topology

                            ┌─────────────┐
                            │   Enqueue   │
                            │   Service   │
                            └──────┬──────┘
                                   │
                     ┌─────────────┼─────────────┐
                     ▼             ▼             ▼
               ┌──────────┐ ┌──────────┐ ┌──────────┐
               │  high    │ │ default  │ │   low    │
               │ priority │ │          │ │ priority │
               └─────┬────┘ └─────┬────┘ └─────┬────┘
                     │            │            │
                     ▼            ▼            ▼
               ┌──────────┐ ┌──────────┐ ┌──────────┐
               │ DLQ-high │ │DLQ-default│ │ DLQ-low │
               └──────────┘ └──────────┘ └──────────┘
                     │            │            │
                     └────────────┼────────────┘
                                  ▼
                           ┌──────────────┐
                           │   Alert on   │
                           │  any DLQ msg │
                           └──────────────┘

Step 4: Job Contract Definition

interface Job<T = unknown> {
  id: string;                         // UUIDv7, sortable by time
  type: string;                       // Discriminator: "send-email" | "generate-report"
  payload: T;                         // Type-safe, JSON-serializable payload
  retryCount: number;                 // Current attempt (0-based)
  maxRetries: number;                 // Max attempts before DLQ
  priority: 'high' | 'medium' | 'low';
  scheduledAt?: string;               // ISO 8601 for delayed execution
  idempotencyKey: string;             // Deduplication: "{job-type}:{entity-id}:{action}"
  tags: string[];                     // Filtering and monitoring
  timeout: number;                    // Per-job timeout in ms
  createdAt: string;                  // ISO 8601
}

// Example: Email sending job
interface SendEmailPayload {
  to: string;
  subject: string;
  templateId: string;
  variables: Record<string, string>;
  tenantId: string;
}

BullMQ job definition:

import { Queue, Worker, Job } from 'bullmq';

const emailQueue = new Queue<SendEmailPayload>('email', {
  connection: { host: 'redis', port: 6379 },
  defaultJobOptions: {
    attempts: 5,
    backoff: { type: 'exponential', delay: 1000 },
    removeOnComplete: { age: 3600 * 24 },
    removeOnFail: { age: 3600 * 24 * 7 },
  },
});

// Enqueue
await emailQueue.add('send-email', payload, {
  priority: 1,
  delay: 0,
  jobId: `send-email:user_123:welcome`, // idempotency key
});

Python Celery:

from celery import Celery

app = Celery('tasks', broker='redis://localhost:6379')

@app.task(
    bind=True,
    max_retries=5,
    default_retry_delay=60,
    acks_late=True,
    reject_on_worker_lost=True,
    queue='default'
)
def send_email(self, to: str, subject: str, template_id: str, variables: dict):
    try:
        email_service.send(to, subject, template_id, variables)
    except RateLimitError as exc:
        raise self.retry(exc=exc, countdown=300)
    except NetworkError as exc:
        raise self.retry(exc=exc, countdown=60)

Go with asynq:

import "github.com/hibiken/asynq"

type EmailPayload struct {
  To      string            `json:"to"`
  Subject string            `json:"subject"`
  TemplateID string         `json:"template_id"`
  Vars    map[string]string `json:"vars"`
}

client := asynq.NewClient(asynq.RedisClientOpt{Addr: "redis:6379"})
payload, _ := json.Marshal(EmailPayload{To: "user@co", Subject: "Welcome"})
task := asynq.NewTask("send:email", payload, asynq.MaxRetry(5), asynq.Queue("email"))
_, err := client.Enqueue(task)

C# Hangfire:

public class SendEmailJob
{
    private readonly IEmailService _emailService;
    public SendEmailJob(IEmailService emailService) => _emailService = emailService;

    [JobDisplayName("Send email to {0}")]
    [AutomaticRetry(Attempts = 5,
    [Queue("default")]
    public async Task ExecuteAsync(string to, string subject, string templateId, Dictionary<string, string> variables)
    {
        await _emailService.SendAsync(to, subject, templateId, variables);
    }
}

BackgroundJob.Enqueue<SendEmailJob>(j => j.ExecuteAsync("user@co", "Welcome", "welcome-template", vars));
BackgroundJob.Schedule<SendEmailJob>(j => j.ExecuteAsync(...), TimeSpan.FromHours(24));

Step 5: Retry Strategy

Exponential backoff with jitter: delay = min(baseDelay * 2^retryCount + jitter, maxDelay).

function calculateDelay(retryCount: number, baseDelay = 1000, maxDelay = 21_600_000): number {
  const delay = Math.min(baseDelay * Math.pow(2, retryCount), maxDelay);
  const jitter = delay * 0.25 * (Math.random() * 2 - 1);
  return Math.round(Math.max(100, delay + jitter));
}

DLQ schema:

interface DeadLetterMessage {
  originalJob: Job;
  errorHistory: Array<{
    attempt: number;
    timestamp: string;
    error: { type: string; message: string; stack?: string };
  }>;
  movedToDLQAt: string;
  dlqReason: string;
}

Step 6: Worker Concurrency

Graceful shutdown:

async function gracefulShutdown(worker: Worker): Promise<void> {
  console.log('Shutting down worker...');
  await worker.close(); // Stop accepting new jobs
  const timeout = setTimeout(() => {
    console.error('Force killing running jobs');
    process.exit(1);
  }, 30000); // 30s timeout

  // Wait for running jobs to complete
  await worker.waitUntilReady();
  clearTimeout(timeout);
  console.log('Worker shutdown complete');
}

process.on('SIGTERM', () => gracefulShutdown(worker));
process.on('SIGINT', () => gracefulShutdown(worker));

Step 7: Idempotency

Same job payload processed twice must produce same result.

async function processJob(job: Job<SendEmailPayload>): Promise<void> {
  const alreadyProcessed = await checkIdempotency(job.idempotencyKey);
  if (alreadyProcessed) { return; }
  await sendEmail(job.payload);
  await markIdempotency(job.idempotencyKey, job.id, 86400); // 24h TTL
}

async function checkIdempotency(key: string): Promise<boolean> {
  return redis.exists(`idempotency:${key}`);
}

async function markIdempotency(key: string, jobId: string, ttl: number): Promise<void> {
  await redis.set(`idempotency:${key}`, jobId, 'EX', ttl);
}

Idempotency key format: {job-type}:{entity-id}:{action} (e.g., send-email:order_123:welcome).

Step 8: Job Scheduling (Cron)

// BullMQ scheduler
import { QueueScheduler } from 'bullmq';

const scheduler = new QueueScheduler('report-generation', {
  connection: { host: 'redis', port: 6379 },
});

// Recurring job every day at 2 AM UTC
await reportQueue.add('daily-report', { type: 'sales' }, {
  repeat: { pattern: '0 2 * * *', tz: 'UTC' },
  jobId: 'daily-sales-report', // Unique ID prevents duplicates
});

Cron best practices:

• Always specify timezone (prefer UTC)
• Prevent overlapping execution (lock or skip if previous still running)

  Expression	Meaning
  0 2 * * *	Daily at 2 AM
  */15 * * * *	Every 15 minutes
  0 0 * * 0	Weekly on Sunday midnight
  0 0 1 * *	Monthly on 1st
  0 9-18 * * 1-5	Every hour 9-6 weekdays

Step 9: Chained Workflows

// BullMQ flow producer
const flow = new FlowProducer({ connection: { host: 'redis', port: 6379 } });

await flow.add({
  name: 'process-order',
  queueName: 'orders',
  data: { orderId: '123' },
  children: [
    {
      name: 'charge-payment',
      queueName: 'payments',
      data: { orderId: '123', amount: 49.99 },
      children: [
        {
          name: 'send-receipt',
          queueName: 'notifications',
          data: { orderId: '123', email: 'user@co' },
        },
        {
          name: 'update-inventory',
          queueName: 'inventory',
          data: { orderId: '123', items: [...] },
        },
      ],
    },
  ],
});

For compensation on failure:

async function processOrder(job: Job): Promise<void> {
  try {
    await chargePayment(job.data.orderId);
    await updateInventory(job.data.orderId);
    await sendReceipt(job.data.orderId);
  } catch (error) {
    // Compensating actions
    await refundPayment(job.data.orderId);
    await restoreInventory(job.data.orderId);
    throw error; // Still move to DLQ for investigation
  }
}

Production Considerations

Job Monitoring Configuration

monitoring:
  metrics:
    - job_success_rate: "p99 > 99%"
    - queue_depth: "alert if > 10000"
    - dlq_count: "alert if > 0"
    - avg_job_duration: "track per job type"
    - worker_pool_saturation: "alert if > 90%"
    - stuck_jobs: "running > 5x expected duration"
  alerts:
    dlq_receives_message: "PagerDuty notification"
    queue_depth_threshold: "Slack notification"
    job_success_rate_below_99: "PagerDuty notification"

Prometheus Metrics

import { Counter, Histogram, Gauge } from 'prom-client';

const jobsProcessed = new Counter({
  name: 'jobs_processed_total',
  help: 'Total jobs processed',
  labelNames: ['queue', 'job_type', 'status'],
});

const jobDuration = new Histogram({
  name: 'job_duration_seconds',
  help: 'Job processing duration',
  labelNames: ['queue', 'job_type'],
  buckets: [0.1, 0.5, 1, 5, 10, 30, 60, 300],
});

const queueDepth = new Gauge({
  name: 'queue_depth',
  help: 'Current queue depth',
  labelNames: ['queue'],
});

const workerUtilization = new Gauge({
  name: 'worker_utilization',
  help: 'Worker pool utilization (0-1)',
  labelNames: ['queue'],
});

Job Dashboard

import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter';
import { ExpressAdapter } from '@bull-board/express';

const serverAdapter = new ExpressAdapter();
createBullBoard({
  queues: [
    new BullMQAdapter(emailQueue),
    new BullMQAdapter(reportQueue),
    new BullMQAdapter(notificationQueue),
  ],
  serverAdapter,
});
app.use('/admin/queues', serverAdapter.getRouter());

Database-Backed Queue (Simple Setup)

CREATE TABLE job_queue (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  type VARCHAR(100) NOT NULL,
  payload JSONB NOT NULL,
  status VARCHAR(20) NOT NULL DEFAULT 'pending',
  priority INT DEFAULT 0,
  scheduled_at TIMESTAMPTZ,
  attempts INT DEFAULT 0,
  max_attempts INT DEFAULT 5,
  last_error TEXT,
  locked_at TIMESTAMPTZ,
  locked_by VARCHAR(100),
  created_at TIMESTAMPTZ DEFAULT NOW(),
  completed_at TIMESTAMPTZ
);

CREATE INDEX idx_job_queue_status ON job_queue(status, scheduled_at, priority);

Anti-Patterns

Anti-Pattern 1: Non-Idempotent Jobs

Problem: Retry causes duplicate charges, emails, or records. Fix: Always check idempotency key before processing. Use deduplication at queue level.

Anti-Pattern 2: Infinite Retries Without Backoff

Problem: Retrying immediately infinitely causes thundering herd on downstream service. Fix: Exponential backoff with jitter. Max retries limit.

Anti-Pattern 3: No Dead Letter Queue

Problem: Failed jobs stay in main queue, block new jobs, cause confusion. Fix: After max retries, move to DLQ. Alert on DLQ message received.

Anti-Pattern 4: Worker Starvation

Problem: One queue with mixed job durations causes short jobs to wait behind long jobs. Fix: Separate queues by duration or use priority tiers.

Anti-Pattern 5: Ignoring Graceful Shutdown

Problem: Abruptly killing workers loses in-flight jobs. Fix: SIGTERM handling with drain and 30s timeout.

Anti-Pattern 6: Blocking the Event Loop

Problem: CPU-intensive job blocks the event loop, starving other jobs in Node.js. Fix: Use worker_threads or separate worker process for CPU-heavy work.

Anti-Pattern 7: Jobs Without Timeouts

Problem: Stuck job holds a worker indefinitely. Fix: Set per-job timeout. Kill and retry after timeout.

Anti-Pattern 8: Synchronous Fallback in Critical Path

Problem: Blocking on a background job from the request handler. Fix: Return 202 Accepted immediately. Use webhook or polling for result.

Security Considerations

Job Payload Security

• Never include secrets, tokens, or PII in job payloads — store references (IDs)
• Encrypt sensitive payload fields with application-level encryption
• Validate payload schema before processing
• Sign job payloads for integrity verification

Access Control

• Queue admin UI requires authentication
• Separate queue management permissions from job processing
• Audit log all enqueue/dequeue/replay operations
• Rate limit enqueue operations per source

Network Security

• Redis/RabbitMQ listeners on internal network only
• TLS connections between workers and queue broker
• mTLS for cross-service job submissions

Comparative Analysis

Queue Backend Comparison

At-Least-Once vs At-Most-Once vs Exactly-Once

Performance Considerations

Job Size Optimization

• Keep payloads <100KB for best performance
• Store large payloads as references (S3 key, DB row ID)
• Use binary serialization for large numeric datasets
• Compress large payloads before enqueuing

Throughput Optimization

• Increase prefetch for short-running jobs
• Separate queues by job duration to prevent head-of-line blocking
• Use batch enqueue for high-volume jobs
• Scale workers horizontally (queue is the bottleneck, not workers)

Redis-Specific Tuning

• Use Redis Cluster for high-throughput scenarios
• Enable lazy Redis connection in worker processes
• Monitor Redis memory and eviction rates
• Pipeline enqueue operations for batch jobs

Rules

• Every job is idempotent — same payload processed twice produces same result
• Job TTL set to prevent stale data processing (24h default, 7d DLQ)
• Workers gracefully drain on shutdown with 30s timeout
• Job payload is JSON-serializable, no circular refs
• Queue depth monitored and alerted at threshold
• Scheduled jobs are cron-expressed in UTC with documented timezone
• Non-recoverable errors go directly to DLQ, no retry
• Job dashboard accessible to on-call engineers
• Set per-job timeout to prevent stuck workers
• Separate queues by job duration to prevent starvation
• Never include secrets in job payloads
• Always alert when DLQ receives a message

References

• references/job-monitoring.md — Background Job Monitoring
• references/job-patterns.md — Background Job Patterns
• references/job-scheduling.md — Background Job Scheduling
• references/job-testing.md — Background Job Testing
• references/job-types.md — Job Types and Queue Patterns
• references/queue-setup.md — Queue Setup and Scheduled Tasks
• references/background-jobs-fundamentals.md — Background Jobs Fundamentals
• references/background-jobs-advanced.md — Background Jobs Advanced Patterns
• references/background-jobs-chaining.md — Job Chaining and Workflow Patterns

Handoff

backend-event-driven for job completion events and chained workflows