Background Jobs & Queue Expert
English | Bahasa Indonesia
English
Description
A deep-dive expert guide for implementing advanced background job queues using BullMQ v5, Trigger.dev v3, and Inngest. This skill focuses purely on the mechanics of robust job processing: delayed jobs, job deduplication, concurrency limits per-tenant, strict idempotency, dead letter queues (DLQ), and job priorities.
Trigger Conditions
- Implementing job queues in a backend application.
- Dealing with flaky third-party APIs that require backoff and retry.
- Designing Dead Letter Queues (DLQ) for failed background tasks.
- Implementing job deduplication and prioritization (e.g., premium user tasks first).
- Offloading heavy compute tasks (video processing, PDF generation, bulk email) from the main request loop.
Advanced Concepts
- Idempotency: Every background job MUST be idempotent. It should be safe to run the exact same job multiple times without creating duplicate side-effects (e.g., charging a customer twice).
- Job Deduplication: Use deterministic job IDs based on payload hash or business logic (e.g.,
invoice_gen_{invoiceId}) to prevent the same job from being enqueued multiple times.
- Dead Letter Queues (DLQ): Jobs that fail repeatedly after all backoff retries must be moved to a DLQ for manual inspection. Never let a poison pill job block the queue.
- Concurrency and Rate Limits: When calling external APIs, restrict worker concurrency to avoid getting rate-limited. E.g., BullMQ
RateLimiter or Trigger.dev concurrency keys.
Bahasa Indonesia
Integrasi Orkestrasi
Terhubung dan mengorkestrasi skill domain yang relevan seperti brainstorming, zero-to-prod-orchestrator, dan project-context-mapper untuk memastikan eksekusi yang kohesif.
Deskripsi
Panduan ahli yang mendalam untuk mengimplementasikan antrean pekerjaan latar belakang (background job queues) menggunakan BullMQ v5, Trigger.dev v3, dan Inngest. Skill ini murni berfokus pada mekanika pemrosesan job yang tangguh: delayed jobs, deduplikasi job, batas konkurensi per-tenant, idempotency yang ketat, dead letter queues (DLQ), dan prioritas job.
Kondisi Pemicu
- Mengimplementasikan job queue pada aplikasi backend.
- Menangani API pihak ketiga yang tidak stabil dan membutuhkan backoff serta retry.
- Merancang Dead Letter Queues (DLQ) untuk tugas yang gagal.
- Mengimplementasikan deduplikasi dan penentuan prioritas job (misalnya, tugas dari pengguna premium didahulukan).
- Memindahkan tugas komputasi berat (pemrosesan video, pembuatan PDF, email massal) dari request loop utama.
Konsep Lanjutan
- Idempotensi (Idempotency): Setiap background job HARUS idempoten. Artinya, job tersebut harus aman dijalankan berkali-kali tanpa menghasilkan efek samping ganda (contoh: menghindari pelanggan ditagih dua kali).
- Deduplikasi Job: Gunakan ID job yang deterministik berdasarkan hash payload atau logika bisnis (contoh:
invoice_gen_{invoiceId}) untuk mencegah job yang sama masuk antrean berulang kali.
- Dead Letter Queues (DLQ): Job yang gagal berulang kali setelah semua percobaan (retry backoff) habis harus dipindahkan ke DLQ untuk diperiksa secara manual. Jangan biarkan job poison pill memblokir antrean.
- Konkurensi dan Batasan Rate: Saat memanggil API eksternal, batasi konkurensi worker untuk menghindari pemblokiran rate limit. Gunakan
RateLimiter di BullMQ atau concurrency keys di Trigger.dev.
Orchestration & Integration
- Connects to
js-backend-expert / go-programming-expert as part of the backend infrastructure.
- Connects to
async-queue-temporal-expert for Saga patterns.
- Connects to
data-telemetry-expert to monitor job queue depth and DLQ alerts.
1---2name: background-jobs-queue-expert3description: Dedicated deep-dive for BullMQ v5, Trigger.dev v3, Inngest, delayed jobs, job deduplication, idempotency, dead letter queues, and job priority / Panduan mendalam untuk BullMQ v5, Trigger.dev v3, Inngest, delayed jobs, deduplikasi job, idempotency, dead letter queue, dan prioritas job.4---56# Background Jobs & Queue Expert78[English](#english) | [Bahasa Indonesia](#bahasa-indonesia)910---1112<a name="english"></a>13## English1415### Description16A deep-dive expert guide for implementing advanced background job queues using BullMQ v5, Trigger.dev v3, and Inngest. This skill focuses purely on the mechanics of robust job processing: delayed jobs, job deduplication, concurrency limits per-tenant, strict idempotency, dead letter queues (DLQ), and job priorities.1718### Trigger Conditions19- Implementing job queues in a backend application.20- Dealing with flaky third-party APIs that require backoff and retry.21- Designing Dead Letter Queues (DLQ) for failed background tasks.22- Implementing job deduplication and prioritization (e.g., premium user tasks first).23- Offloading heavy compute tasks (video processing, PDF generation, bulk email) from the main request loop.2425### Advanced Concepts26271. **Idempotency**: Every background job MUST be idempotent. It should be safe to run the exact same job multiple times without creating duplicate side-effects (e.g., charging a customer twice).282. **Job Deduplication**: Use deterministic job IDs based on payload hash or business logic (e.g., `invoice_gen_{invoiceId}`) to prevent the same job from being enqueued multiple times.293. **Dead Letter Queues (DLQ)**: Jobs that fail repeatedly after all backoff retries must be moved to a DLQ for manual inspection. Never let a poison pill job block the queue.304. **Concurrency and Rate Limits**: When calling external APIs, restrict worker concurrency to avoid getting rate-limited. E.g., BullMQ `RateLimiter` or Trigger.dev concurrency keys.3132---3334<a name="bahasa-indonesia"></a>35## Bahasa Indonesia3637### Integrasi Orkestrasi38Terhubung dan mengorkestrasi skill domain yang relevan seperti `brainstorming`, `zero-to-prod-orchestrator`, dan `project-context-mapper` untuk memastikan eksekusi yang kohesif.3940### Deskripsi41Panduan ahli yang mendalam untuk mengimplementasikan antrean pekerjaan latar belakang (background job queues) menggunakan BullMQ v5, Trigger.dev v3, dan Inngest. Skill ini murni berfokus pada mekanika pemrosesan job yang tangguh: *delayed jobs*, deduplikasi job, batas konkurensi per-tenant, *idempotency* yang ketat, *dead letter queues* (DLQ), dan prioritas job.4243### Kondisi Pemicu44- Mengimplementasikan job queue pada aplikasi backend.45- Menangani API pihak ketiga yang tidak stabil dan membutuhkan *backoff* serta *retry*.46- Merancang Dead Letter Queues (DLQ) untuk tugas yang gagal.47- Mengimplementasikan deduplikasi dan penentuan prioritas job (misalnya, tugas dari pengguna premium didahulukan).48- Memindahkan tugas komputasi berat (pemrosesan video, pembuatan PDF, email massal) dari *request loop* utama.4950### Konsep Lanjutan51521. **Idempotensi (Idempotency)**: Setiap background job HARUS idempoten. Artinya, job tersebut harus aman dijalankan berkali-kali tanpa menghasilkan efek samping ganda (contoh: menghindari pelanggan ditagih dua kali).532. **Deduplikasi Job**: Gunakan ID job yang deterministik berdasarkan hash payload atau logika bisnis (contoh: `invoice_gen_{invoiceId}`) untuk mencegah job yang sama masuk antrean berulang kali.543. **Dead Letter Queues (DLQ)**: Job yang gagal berulang kali setelah semua percobaan (retry backoff) habis harus dipindahkan ke DLQ untuk diperiksa secara manual. Jangan biarkan job *poison pill* memblokir antrean.554. **Konkurensi dan Batasan Rate**: Saat memanggil API eksternal, batasi konkurensi *worker* untuk menghindari pemblokiran *rate limit*. Gunakan `RateLimiter` di BullMQ atau concurrency keys di Trigger.dev.5657## Orchestration & Integration58- Connects to `js-backend-expert` / `go-programming-expert` as part of the backend infrastructure.59- Connects to `async-queue-temporal-expert` for Saga patterns.60- Connects to `data-telemetry-expert` to monitor job queue depth and DLQ alerts.
roedyrustam