Celery Patterns

Celery Patterns for Django

Core Philosophy

• Idempotent tasks: Running a task twice must produce the same result
• Pass IDs, not objects: Arguments must be JSON-serializable
• Always handle failures: Log errors, never swallow exceptions silently
• Proper retry strategies: Use exponential backoff for external services

Task Design

Structure

• Place tasks in apps/<app>/tasks.py
• Use @shared_task for reusable tasks across projects
• Use bind=True when needing access to task instance (retries, task ID, state updates)
• Add type hints to task signatures
• Log task start, completion, and failures

Arguments

• Pass model primary keys, not model instances
• Keep arguments simple and serializable (str, int, dict, list)
• Validate that referenced objects exist at task start

Retry Strategies

When to Use Each Approach

• Fixed delay: Internal operations with predictable recovery (database locks)
• Exponential backoff: External APIs that may rate-limit or have variable recovery
• No retry: Validation errors, business logic failures, permanent errors

Configuration

• Set max_retries based on acceptable total wait time
• Use retry_jitter=True to prevent thundering herd
• Set retry_backoff_max to cap maximum wait time
• Use autoretry_for tuple for automatic retry on specific exceptions

Idempotency Patterns

Check-Before-Process

Query current state before processing; skip if already complete

Status Field Tracking

Use status transitions (pending → processing → complete/failed) with select_for_update() for race condition safety

Unique Constraints

Use database constraints to prevent duplicate processing

Periodic Tasks (Beat)

• Configure schedules in config/celery.py using beat_schedule
• Use crontab() for time-based schedules
• Use float values for interval-based schedules (seconds)
• Keep periodic tasks lightweight; spawn subtasks for heavy work

Anti-Patterns to Avoid

• Passing model instances instead of IDs
• Non-idempotent operations (incrementing without checks)
• Silent exception handling (bare except: pass)
• Missing logging for task lifecycle
• Long-running tasks without progress updates
• Retry on permanent failures (validation, business logic errors)

Commands

• Worker: uv run celery -A config worker -l info
• Beat: uv run celery -A config beat -l info
• Monitoring: uv run celery -A config flower