Async Jobs
Background task processing with Celery, ARQ, Redis and Temporal. This skill is a wrapper, not a
manual: Celery and ARQ document their own product well, so what lives here is our delta, the
thresholds, working config, ordering constraints and tool-choice rules we picked. Product
mechanics are linked, not restated.
Start with Read("references/ork-delta.md").
Quick Reference
| Topic |
Where our part lives |
| Configuration |
references/celery-config.md, rules/jobs-task-queue.md |
| Task Routing |
references/ork-delta.md (queue taxonomy, prefetch tiers, Redis priority) |
| Canvas Workflows |
rules/celery-canvas.md |
| Retry Strategies |
references/ork-delta.md (backoff cap, idempotency layers, lock TTLs) |
| Scheduling |
rules/jobs-scheduling.md, references/ork-delta.md (beat process model) |
| Monitoring |
references/ork-delta.md (alert thresholds, histogram buckets) |
| Result Backends |
rules/jobs-monitoring.md, references/ork-delta.md (return contract) |
| ARQ Patterns |
rules/jobs-task-queue.md, references/ork-delta.md (budgets, pool ownership) |
| Temporal Workflows |
rules/temporal-workflows.md |
| Temporal Activities |
rules/temporal-activities.md |
10 topic areas, 6 rule files in rules/, house delta in references/ork-delta.md.
Quick Start
@app.task(bind=True, max_retries=3, default_retry_delay=60)
def process_payment(self, order_id: str):
try:
return gateway.charge(order_id)
except TransientError as exc:
raise self.retry(exc=exc, countdown=2 ** self.request.retries * 60)
Load more examples: Read("references/quick-start-examples.md") for Celery
retry task and ARQ/FastAPI integration patterns.
Upstream coverage (do not restate)
Fetch these when you need product mechanics. The right-hand column is the part we keep, because
it is a house threshold, a working config or an ordering constraint that upstream cannot know.
| Topic |
First-party source |
House subset stays in |
| Celery settings, serializers, time limits, worker flags |
https://docs.celeryq.dev/en/stable/userguide/configuration.html and .../optimizing.html |
references/celery-config.md, rules/jobs-task-queue.md |
| Queue declarations, router classes, Redis priority mechanics |
https://docs.celeryq.dev/en/stable/userguide/routing.html |
references/ork-delta.md |
| chain / group / chord / signature semantics |
https://docs.celeryq.dev/en/stable/userguide/canvas.html |
rules/celery-canvas.md keeps the house canvas subset. Its si()-in-chords guidance is UNVERIFIED and contested: confirm the argument-passing behaviour against the upstream canvas page before relying on it |
autoretry_for, retry_backoff, Reject, task base classes |
https://docs.celeryq.dev/en/stable/userguide/tasks.html |
references/ork-delta.md, rules/jobs-task-queue.md |
| Beat schedules, crontab syntax, DatabaseScheduler |
https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html and https://django-celery-beat.readthedocs.io/en/latest/ |
rules/jobs-scheduling.md keeps our beat_schedule shapes; references/ork-delta.md keeps the process model |
Flower flags, inspect, signal names |
https://docs.celeryq.dev/en/stable/userguide/monitoring.html, https://docs.celeryq.dev/en/stable/userguide/signals.html, https://flower.readthedocs.io/en/latest/config.html |
references/ork-delta.md |
Result backend, AsyncResult, custom states |
https://docs.celeryq.dev/en/stable/userguide/configuration.html |
rules/jobs-monitoring.md keeps our status endpoints and update_state() usage |
Per-task rate_limit, control.rate_limit, Redis Lua |
https://docs.celeryq.dev/en/stable/userguide/workers.html, https://redis.io/docs/latest/develop/programmability/eval-intro/ |
references/ork-delta.md |
ARQ WorkerSettings, enqueue_job, _defer_by / _defer_until, Job status |
https://arq-docs.helpmanual.io/ |
rules/jobs-task-queue.md keeps the worker skeleton; references/ork-delta.md keeps the budgets |
| FastAPI lifespan and dependency wiring |
https://fastapi.tiangolo.com/advanced/events/ |
references/ork-delta.md |
Distributed locks with SET NX EX |
https://redis.io/docs/latest/commands/set/ |
references/ork-delta.md |
Configuration
Load: Read("references/celery-config.md").
| Decision |
Recommendation |
| Serializer |
JSON (never pickle) |
| Ack mode |
Late ack (task_acks_late=True) |
| Prefetch |
1 for fair, 4-8 for throughput |
| Time limit |
soft < hard (540 / 600) |
| Timezone |
UTC always |
Task Routing
| Decision |
Recommendation |
| Queue count |
5: critical / high / default / low / bulk |
| Priority levels |
0-9, with all four Redis priority switches set together |
| Worker assignment |
Dedicated worker per queue |
| Prefetch |
1 critical, 2 high, 4 default, 8 low/bulk |
| Routing |
Router class once past 5 routing rules |
Canvas Workflows
Load: Read("rules/celery-canvas.md").
| Decision |
Recommendation |
| Sequential |
Chain with s() |
| Parallel |
Group for independent tasks |
| Fan-in |
Chord (all header tasks must succeed for the body to run) |
| Ignore input |
Use si() immutable signature |
| Error in chain |
Reject stops the chain, retry continues it |
| Partial failures |
Return an error dict from chord header tasks |
Retry Strategies
| Decision |
Recommendation |
| Retry delay |
Exponential backoff, jitter on, capped at 600s |
| Max retries |
3-5 for transient, 0 for permanent |
| Idempotency |
Redis marker (86400s TTL) plus the vendor idempotency key |
| Failed tasks |
DLQ for manual review |
| Singleton |
Redis lock with a TTL longer than the hard time limit |
Scheduling
Load: Read("rules/jobs-scheduling.md").
| Decision |
Recommendation |
| Schedule type |
Crontab for time-based, float interval for frequency |
| Dynamic |
DatabaseScheduler (django-celery-beat) |
| Overlap |
Redis lock, 3600s default and 7200s for long jobs |
| Beat process |
Separate process; embedded --beat is development only |
| Timezone |
UTC always |
Monitoring
| Decision |
Recommendation |
| Dashboard |
Flower with persistent storage |
| Metrics |
Prometheus wired to task_prerun / task_postrun / task_failure |
| Health |
Broker reachable, at least one worker, queue depths |
| Alerting |
critical > 100, default > 5000, workers < 1 |
| Autoscale |
Queue depth > 500 |
Result Backends
Load: Read("rules/jobs-monitoring.md").
| Decision |
Recommendation |
| Status storage |
Redis result backend, status and small JSON only |
| Large results |
S3 or database, task returns a reference dict |
| Progress |
Custom states with update_state() |
| Result query |
AsyncResult with state checks |
ARQ Patterns
Load: Read("rules/jobs-task-queue.md").
| Decision |
Recommendation |
| Simple async |
ARQ (native async), max_jobs=10, job_timeout=300 |
| Pool ownership |
FastAPI lifespan, never a per-request create_pool |
| Complex workflows |
Celery (chains, chords, DLQ, per-task rate limits) |
| In-process quick |
FastAPI BackgroundTasks, under 30s, non-critical only |
| LLM workflows |
LangGraph, not Celery |
Tool Selection
Load: Read("references/quick-start-examples.md") for the full tool
comparison table (ARQ, Celery, RQ, Dramatiq, FastAPI BackgroundTasks).
Anti-Patterns (FORBIDDEN)
Load details: Read("references/anti-patterns.md") for the full list.
Key rules: never run long tasks in request handlers, never block on results inside tasks, never
store large results in Redis, always use idempotency for retried tasks.
Temporal Workflows
Load: Read("rules/temporal-workflows.md").
| Decision |
Recommendation |
| Workflow ID |
Business-meaningful, idempotent |
| Determinism |
Use workflow.random(), workflow.now() |
| I/O |
Always via activities, never directly |
Temporal Activities
Load: Read("rules/temporal-activities.md").
| Decision |
Recommendation |
| Activity timeout |
start_to_close for most cases |
| Error handling |
Non-retryable for business errors |
| Testing |
WorkflowEnvironment.start_local() for integration tests |
Related Skills
ork:python-backend - FastAPI, asyncio, SQLAlchemy patternsork:langgraph - LangGraph workflow patterns (use for LLM workflows, not Celery)ork:distributed-systems - Resilience patterns, circuit breakersork:monitoring-observability - Metrics and alerting
Capability Details
Load details: Read("references/capability-details.md") for the keyword index
and problem-to-capability mapping.
1---2name: async-jobs3description: Async job processing patterns for background tasks, Celery workflows, task scheduling, retry strategies, and distributed task execution. Use when implementing background job processing, task queues, or scheduled task systems.4license: MIT5---6
7# Async Jobs
8
9Background task processing with Celery, ARQ, Redis and Temporal. This skill is a wrapper, not a
10manual: Celery and ARQ document their own product well, so what lives here is our delta, the
11thresholds, working config, ordering constraints and tool-choice rules we picked. Product
12mechanics are linked, not restated.
13
14Start with `Read("references/ork-delta.md")`.
15
16## Quick Reference
17
18| Topic | Where our part lives |
19|-------|----------------------|
20| [Configuration](#configuration) | `references/celery-config.md`, `rules/jobs-task-queue.md` |
21| [Task Routing](#task-routing) | `references/ork-delta.md` (queue taxonomy, prefetch tiers, Redis priority) |
22| [Canvas Workflows](#canvas-workflows) | `rules/celery-canvas.md` |
23| [Retry Strategies](#retry-strategies) | `references/ork-delta.md` (backoff cap, idempotency layers, lock TTLs) |
24| [Scheduling](#scheduling) | `rules/jobs-scheduling.md`, `references/ork-delta.md` (beat process model) |
25| [Monitoring](#monitoring) | `references/ork-delta.md` (alert thresholds, histogram buckets) |
26| [Result Backends](#result-backends) | `rules/jobs-monitoring.md`, `references/ork-delta.md` (return contract) |
27| [ARQ Patterns](#arq-patterns) | `rules/jobs-task-queue.md`, `references/ork-delta.md` (budgets, pool ownership) |
28| [Temporal Workflows](#temporal-workflows) | `rules/temporal-workflows.md` |
29| [Temporal Activities](#temporal-activities) | `rules/temporal-activities.md` |
30
3110 topic areas, 6 rule files in `rules/`, house delta in `references/ork-delta.md`.
32
33## Quick Start
34
35```python
36@app.task(bind=True, max_retries=3, default_retry_delay=60)
37def process_payment(self, order_id: str):
38 try:
39 return gateway.charge(order_id)
40 except TransientError as exc:
41 raise self.retry(exc=exc, countdown=2 ** self.request.retries * 60)
42```
43
44Load more examples: `Read("references/quick-start-examples.md")` for Celery
45retry task and ARQ/FastAPI integration patterns.
46
47## Upstream coverage (do not restate)
48
49Fetch these when you need product mechanics. The right-hand column is the part we keep, because
50it is a house threshold, a working config or an ordering constraint that upstream cannot know.
51
52| Topic | First-party source | House subset stays in |
53|-------|--------------------|-----------------------|
54| Celery settings, serializers, time limits, worker flags | https://docs.celeryq.dev/en/stable/userguide/configuration.html and .../optimizing.html | `references/celery-config.md`, `rules/jobs-task-queue.md` |
55| Queue declarations, router classes, Redis priority mechanics | https://docs.celeryq.dev/en/stable/userguide/routing.html | `references/ork-delta.md` |
56| chain / group / chord / signature semantics | https://docs.celeryq.dev/en/stable/userguide/canvas.html | `rules/celery-canvas.md` keeps the house canvas subset. Its `si()`-in-chords guidance is UNVERIFIED and contested: confirm the argument-passing behaviour against the upstream canvas page before relying on it |
57| `autoretry_for`, `retry_backoff`, `Reject`, task base classes | https://docs.celeryq.dev/en/stable/userguide/tasks.html | `references/ork-delta.md`, `rules/jobs-task-queue.md` |
58| Beat schedules, crontab syntax, DatabaseScheduler | https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html and https://django-celery-beat.readthedocs.io/en/latest/ | `rules/jobs-scheduling.md` keeps our `beat_schedule` shapes; `references/ork-delta.md` keeps the process model |
59| Flower flags, `inspect`, signal names | https://docs.celeryq.dev/en/stable/userguide/monitoring.html, https://docs.celeryq.dev/en/stable/userguide/signals.html, https://flower.readthedocs.io/en/latest/config.html | `references/ork-delta.md` |
60| Result backend, `AsyncResult`, custom states | https://docs.celeryq.dev/en/stable/userguide/configuration.html | `rules/jobs-monitoring.md` keeps our status endpoints and `update_state()` usage |
61| Per-task `rate_limit`, `control.rate_limit`, Redis Lua | https://docs.celeryq.dev/en/stable/userguide/workers.html, https://redis.io/docs/latest/develop/programmability/eval-intro/ | `references/ork-delta.md` |
62| ARQ `WorkerSettings`, `enqueue_job`, `_defer_by` / `_defer_until`, `Job` status | https://arq-docs.helpmanual.io/ | `rules/jobs-task-queue.md` keeps the worker skeleton; `references/ork-delta.md` keeps the budgets |
63| FastAPI lifespan and dependency wiring | https://fastapi.tiangolo.com/advanced/events/ | `references/ork-delta.md` |
64| Distributed locks with `SET NX EX` | https://redis.io/docs/latest/commands/set/ | `references/ork-delta.md` |
65
66## Configuration
67
68Load: `Read("references/celery-config.md")`.
69
70| Decision | Recommendation |
71|----------|----------------|
72| Serializer | JSON (never pickle) |
73| Ack mode | Late ack (`task_acks_late=True`) |
74| Prefetch | 1 for fair, 4-8 for throughput |
75| Time limit | soft < hard (540 / 600) |
76| Timezone | UTC always |
77
78## Task Routing
79
80| Decision | Recommendation |
81|----------|----------------|
82| Queue count | 5: critical / high / default / low / bulk |
83| Priority levels | 0-9, with all four Redis priority switches set together |
84| Worker assignment | Dedicated worker per queue |
85| Prefetch | 1 critical, 2 high, 4 default, 8 low/bulk |
86| Routing | Router class once past 5 routing rules |
87
88## Canvas Workflows
89
90Load: `Read("rules/celery-canvas.md")`.
91
92| Decision | Recommendation |
93|----------|----------------|
94| Sequential | Chain with `s()` |
95| Parallel | Group for independent tasks |
96| Fan-in | Chord (all header tasks must succeed for the body to run) |
97| Ignore input | Use `si()` immutable signature |
98| Error in chain | `Reject` stops the chain, `retry` continues it |
99| Partial failures | Return an error dict from chord header tasks |
100
101## Retry Strategies
102
103| Decision | Recommendation |
104|----------|----------------|
105| Retry delay | Exponential backoff, jitter on, capped at 600s |
106| Max retries | 3-5 for transient, 0 for permanent |
107| Idempotency | Redis marker (86400s TTL) plus the vendor idempotency key |
108| Failed tasks | DLQ for manual review |
109| Singleton | Redis lock with a TTL longer than the hard time limit |
110
111## Scheduling
112
113Load: `Read("rules/jobs-scheduling.md")`.
114
115| Decision | Recommendation |
116|----------|----------------|
117| Schedule type | Crontab for time-based, float interval for frequency |
118| Dynamic | DatabaseScheduler (`django-celery-beat`) |
119| Overlap | Redis lock, 3600s default and 7200s for long jobs |
120| Beat process | Separate process; embedded `--beat` is development only |
121| Timezone | UTC always |
122
123## Monitoring
124
125| Decision | Recommendation |
126|----------|----------------|
127| Dashboard | Flower with persistent storage |
128| Metrics | Prometheus wired to `task_prerun` / `task_postrun` / `task_failure` |
129| Health | Broker reachable, at least one worker, queue depths |
130| Alerting | critical > 100, default > 5000, workers < 1 |
131| Autoscale | Queue depth > 500 |
132
133## Result Backends
134
135Load: `Read("rules/jobs-monitoring.md")`.
136
137| Decision | Recommendation |
138|----------|----------------|
139| Status storage | Redis result backend, status and small JSON only |
140| Large results | S3 or database, task returns a reference dict |
141| Progress | Custom states with `update_state()` |
142| Result query | `AsyncResult` with state checks |
143
144## ARQ Patterns
145
146Load: `Read("rules/jobs-task-queue.md")`.
147
148| Decision | Recommendation |
149|----------|----------------|
150| Simple async | ARQ (native async), `max_jobs=10`, `job_timeout=300` |
151| Pool ownership | FastAPI lifespan, never a per-request `create_pool` |
152| Complex workflows | Celery (chains, chords, DLQ, per-task rate limits) |
153| In-process quick | FastAPI BackgroundTasks, under 30s, non-critical only |
154| LLM workflows | LangGraph, not Celery |
155
156## Tool Selection
157
158Load: `Read("references/quick-start-examples.md")` for the full tool
159comparison table (ARQ, Celery, RQ, Dramatiq, FastAPI BackgroundTasks).
160
161## Anti-Patterns (FORBIDDEN)
162
163Load details: `Read("references/anti-patterns.md")` for the full list.
164
165Key rules: never run long tasks in request handlers, never block on results inside tasks, never
166store large results in Redis, always use idempotency for retried tasks.
167
168## Temporal Workflows
169
170Load: `Read("rules/temporal-workflows.md")`.
171
172| Decision | Recommendation |
173|----------|----------------|
174| Workflow ID | Business-meaningful, idempotent |
175| Determinism | Use `workflow.random()`, `workflow.now()` |
176| I/O | Always via activities, never directly |
177
178## Temporal Activities
179
180Load: `Read("rules/temporal-activities.md")`.
181
182| Decision | Recommendation |
183|----------|----------------|
184| Activity timeout | `start_to_close` for most cases |
185| Error handling | Non-retryable for business errors |
186| Testing | `WorkflowEnvironment.start_local()` for integration tests |
187
188## Related Skills
189
190- `ork:python-backend` - FastAPI, asyncio, SQLAlchemy patterns
191- `ork:langgraph` - LangGraph workflow patterns (use for LLM workflows, not Celery)
192- `ork:distributed-systems` - Resilience patterns, circuit breakers
193- `ork:monitoring-observability` - Metrics and alerting
194
195## Capability Details
196
197Load details: `Read("references/capability-details.md")` for the keyword index
198and problem-to-capability mapping.
yonatangross