Celery Knowledge Patch

Celery Knowledge Patch

Use this skill when upgrading, configuring, operating, or testing Celery and the work can depend on recent worker, queue, broker, backend, scheduling, or task behavior.

How to use this skill

1. Inspect the project's Celery version and its pinned Python, Kombu, Billiard, broker-client, web-framework, and backend dependencies.
2. Identify the affected surface: compatibility, brokers and backends, queues and publishing, scheduling and testing, tasks and workflows, or workers and shutdown.
3. Read the matching reference file before recommending configuration or code.
4. Prefer the project's manifest, lockfile, configuration, code, and observed behavior when they disagree with general guidance.
5. Treat settings that change delivery, acknowledgement, shutdown, or schema creation semantics as rollout decisions, not mechanical renames.
6. Verify the result with a representative worker and the actual broker or result backend when practical.

Reference index

Upgrade decisions first

Check runtime floors before changing behavior

Do not begin an upgrade by copying new settings into an older environment. First check the Python implementation and version, then Kombu and Billiard. Also check redis-py, Django, SQLAlchemy, and the SQS transport dependency when those integrations are present.

The Python floor moves between the relevant release lines. A deployment that runs on the oldest supported Python in one line may need a Python upgrade before moving to the next. See Compatibility and upgrade decisions for the exact matrices.

Account for the SQS dependency reversal

The SQS transport's HTTP dependency changes direction across these release lines. Rebuild images and dependency allowlists instead of assuming the prior replacement remains valid. Confirm that the required native build packages and runtime library are present before deploying SQS workers.

Treat the support window as an operating constraint

Do not describe the 5.x line as LTS. Plan migration work with the documented end of 5.x support in mind, and do not propose Celery 4.x as a supported fallback.

Review changed lifecycle semantics

Several settings alter when work is rejected, requeued, allowed to finish, or cancelled. Before rollout, test hard time limits, cold shutdown, soft shutdown, idle workers, and the signals sent by the deployment platform.

High-value configuration

Bound graceful termination

Use a positive worker_soft_shutdown_timeout to give active tasks a bounded completion interval before cold shutdown. The feature is disabled when the timeout is zero. If idle workers must also enter soft shutdown, enable worker_enable_soft_shutdown_on_idle explicitly.

This is especially important with visibility-timeout brokers, where abrupt termination can delay redelivery. Match the timeout to orchestrator grace periods and task behavior. See Workers and shutdown.

Align container signals with intended shutdown

If the runtime sends SIGTERM but the desired worker path is the SIGQUIT path, set the supported REMAP_SIGTERM environment variable. Document this at the container or service boundary because it changes the meaning of the runtime's default stop signal.

Cap in-memory ETA work when necessary

Set worker_eta_task_limit when a worker can receive a large number of ETA or countdown tasks. The default is unlimited, so the setting only provides protection after an explicit finite value is chosen. Pair the limit with monitoring and publisher behavior; see Queues and publishing.

Choose queue types deliberately

The default task queue remains classic unless configured otherwise. Native delayed delivery uses quorum queues by default, and workers can detect quorum queues automatically. Decide separately:

• the default task queue type;
• the delayed-delivery queue type;
• whether worker quorum detection is enabled; and
• the queue and exchange types used for auto-created missing queues.

Read Queues and publishing before mixing classic, quorum, and auto-created queues.

Make backend table ownership explicit

The database result backend eagerly creates its tables during setup by default. Disable create_tables_at_setup when migrations or another schema owner must control table creation. See Brokers and result backends.

Task authoring quick reference

Pydantic-aware tasks

Set pydantic=True on a task to validate and convert annotated Pydantic input before invocation and serialize an annotated Pydantic result. Use the additional task options to select strict validation, provide validation context, or customize model dumping.

Type annotations remain part of the contract. Account for optional values and generic annotations rather than inserting manual conversion that duplicates the task wrapper. Full examples are in Tasks and workflows.

Single-item groups in chains

A single-item group chained into another signature unrolls to the contained signature. The next task receives the scalar task result, not a one-element list. Audit downstream tasks that normalize or index a list solely for this case.

Parse standard crontab text directly

Use crontab.from_string() for a standard five-field expression. Month names can be supplied to crontab where that improves configuration readability. Keep validation at configuration boundaries; details are in Scheduling and testing.

Broker and backend quick reference

Redis resilience and identity

Temporary Redis result-backend failures can be classified as safe to retry, and the broker stack includes improved disconnection recovery. Configure retry policy deliberately; resilience does not make every exception safe.

For managed Redis authentication, use redis_backend_credential_provider. Use redis_client_name to label backend connections for monitoring. Avoid embedding credentials in diagnostic output even though delayed-delivery broker URL logging sanitizes passwords.

Bound publisher I/O

Pass timeout and confirm_timeout through apply_async() when callers need bounds on broker I/O and publisher-confirm waits. Choose values with broker latency and retry behavior in mind.

Select the correct backend connection form

The Google Cloud Pub/Sub broker uses the gcpubsub extra and a project URL. The DynamoDB result backend accepts a remote local-service host rather than requiring localhost. Azure Blob result storage can use managed credentials. Use the focused examples in Brokers and result backends.

Worker behavior quick reference

Terminate gevent tasks intentionally

The gevent pool can terminate running requests and jobs, so a revoke with terminate=True can now stop active work. Treat forced termination as a last-resort operation because task code may not complete cleanup.

Validate hard-timeout acknowledgement policy

When task_acks_on_failure_or_timeout is false, a task that exceeds its hard time limit is rejected. Test redelivery and dead-letter behavior with the actual broker before changing this setting.

Avoid inherited Django connection pools

Django psycopg3 pools are closed before worker forks to prevent inherited pools from timing out. If pool errors persist, verify that connection setup occurs in the correct post-fork lifecycle and inspect application-specific pool creation.

Do not infer cold-shutdown timeout failures

Cold shutdown skips timeout-failure handling. During shutdown incidents, distinguish tasks that complete, are requeued, or are actually timed out; do not classify every interrupted task as a timeout failure.

Verification checklist

• Confirm runtime and dependency floors from the deployed environment, not only from a developer lockfile.
• Inspect the effective queue declaration and exchange type on the broker.
• Publish a task with the configured timeout and confirm policy.
• Exercise ETA or countdown tasks under the configured memory cap.
• Test soft, cold, and signal-remapped shutdown with active and idle workers.
• Verify hard-timeout rejection and broker redelivery behavior.
• Start workers against each configured result backend and authentication mode.
• Run a Pydantic task with valid, invalid, optional, and returned model data.
• Exercise chained groups with both one item and multiple items.
• Run scheduling and worker-hostname tests through the project's actual test fixtures.