Celery Knowledge Patch

Celery Knowledge Patch

Load this skill when maintaining Celery applications, workers, brokers, result backends, schedules, or task workflows. Start with the quick reference, then open the topic file that matches the work at hand.

Reference index

Check compatibility before upgrading

Runtime requirements changed

• For 5.5, use CPython 3.8–3.13 or PyPy 3.10+, Kombu 5.5 or newer, redis-py 4.5.2 or newer, Billiard 4.2.1 or newer, and Django 2.2.28 or newer.
• For 5.6, use CPython 3.9–3.13 or PyPy 3.11, Kombu 5.6 or newer, and Billiard 4.2.4 or newer.
• SQLAlchemy 1.4 and 2.0 are supported.
• Celery 4.x is unsupported. Celery 5.x is not an LTS line and is supported only until Celery 6.x.

See runtime-and-lifecycle.md before changing Python, framework, or transport dependencies.

SQS uses pycurl again

Celery 5.5 replaced the pycurl dependency with urllib3, but the 5.6 SQS transport reverses that change and uses pycurl again. Ensure 5.6 SQS worker images and deployment environments provide the restored dependency.

Singleton groups unroll

A one-item group chained with | unrolls to its contained signature. The downstream task receives the item result, not a one-element list:

workflow = group(add.s(1, 1)) | consume.s()

Here, consume receives 2. Audit downstream tasks that assumed a list.

Hard timeouts can reject tasks

If a task exceeds its hard time limit while task_acks_on_failure_or_timeout is False, the worker rejects it:

task_acks_on_failure_or_timeout = False

Configure queues and delayed tasks

Use quorum queues deliberately

The default task queue type remains classic. To make the default task queue a quorum queue, configure:

task_default_queue_type = "quorum"

Quorum queues support ETA tasks. Workers detect them by default and automatically enable native delayed delivery. The delayed-delivery queue type defaults to quorum:

broker_native_delayed_delivery_queue_type = "quorum"
worker_detect_quorum_queues = True

For queues Celery creates on demand, choose their queue type separately:

task_create_missing_queue_type = "quorum"

Use task_create_missing_queue_exchange_type when the auto-created queue also needs an explicit exchange type.

Cap in-memory ETA tasks

Workers otherwise hold an unlimited number of ETA and countdown tasks in memory. Set a cap for workloads that can accumulate many scheduled messages:

worker_eta_task_limit = 1000

See brokers-and-queues.md for transport, publishing, and credential details.

Make shutdown behavior explicit

Allow a soft-shutdown interval

Soft shutdown gives active tasks a bounded interval to finish before cold shutdown cancels them. It is disabled by default because its timeout is 0.0:

worker_soft_shutdown_timeout = 30.0
worker_enable_soft_shutdown_on_idle = True

Idle-worker soft shutdown defaults to False. Enabling both settings is especially useful with visibility-timeout brokers such as Redis and SQS.

Remap container termination

To make SIGTERM follow the SIGQUIT shutdown path, set:

export REMAP_SIGTERM="SIGQUIT"

This is useful when a container runtime sends SIGTERM by default. During cold shutdown, workers skip timeout-failure handling so tasks are not marked with spurious timeout failures and can complete or be requeued correctly.

Use typed Pydantic tasks

Set pydantic=True on a task to convert type-hinted Pydantic arguments before the task runs and dump a returned Pydantic model to a dictionary:

from celery import Celery
from pydantic import BaseModel

app = Celery("tasks")

class Input(BaseModel):
    value: int

@app.task(pydantic=True)
def double(arg: Input) -> Input:
    return Input(value=arg.value * 2)

Optional and generic annotations are recognized. Use:

• pydantic_strict for strict validation.
• pydantic_context to pass validation context.
• pydantic_dump_kwargs to customize result serialization.

See tasks-and-scheduling.md for optional models, crontab parsing, publishing timeouts, and task termination.

Choose backend initialization and credentials

The database result backend creates its tables during backend initialization because create_tables_at_setup defaults to True. Set it to False to keep lazy first-use creation when another system manages the schema.

Redis result backends support provider-based authentication through redis_backend_credential_provider, including AWS ElastiCache IAM authentication. Set redis_client_name to label backend connections for Redis monitoring.

See result-backends.md for Redis recovery, remote local-DynamoDB endpoints, and Azure managed credentials.