# Celery Development Best Practices > Comprehensive Celery distributed task queue framework with 60+ prioritized rules across configuration, task execution, Canvas workflows, monitoring, security, performance, routing, periodic tasks, serialization, worker management, and advanced patterns. Use when implementing background tasks, periodic jobs, or distributed processing with Celery. - Skill: `gnsubrahmanyam/celery-development-best-practices` (Agent Skill, multi-file: 28 files) - Install (CLI): `npx skillmds@latest add gnsubrahmanyam/celery-development-best-practices` - Raw SKILL.md: https://api.skillmd.com/api/skills/gnsubrahmanyam/celery-development-best-practices/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Security - Author: GNSubrahmanyam (https://skillmd.com/u/gnsubrahmanyam) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/gnsubrahmanyam/celery-development-best-practices --- # Celery Development Best Practices Comprehensive background task processing framework with 60+ prioritized rules across 21 categories for building reliable, scalable distributed task systems with Celery. ## When to Apply Reference these guidelines when: - Implementing background task processing - Setting up distributed job queues - Configuring periodic tasks with Celery Beat - Optimizing task performance and reliability - Monitoring and debugging Celery applications - Implementing task routing and result handling - Managing Celery workers and brokers - Ensuring security in distributed task environments - Designing complex workflows with Canvas primitives - Handling serialization and data transmission - Deploying Celery in production environments ## Rule Categories by Priority | Priority | Category | Rules | Impact | Prefix | | --- | --- | --- | --- | --- | | 1 | Configuration & Setup | 2 | CRITICAL | `config-` | | 2 | Task Definition & Execution | 3 | CRITICAL | `task-` | | 3 | Error Handling & Reliability | 1 | HIGH | `error-` | | 4 | Canvas: Designing Work-flows | 4 | HIGH | `canvas-` | | 5 | Monitoring & Logging | 2 | HIGH | `monitoring-` / `perf-` | | 6 | Performance & Scaling | 2 | MEDIUM-HIGH | `perf-` | | 7 | Security | 1 | MEDIUM-HIGH | `security-` | | 8 | Result Backends | 2 | MEDIUM-HIGH | `result-` | | 9 | Routing & Queues | 1 | MEDIUM | `routing-` | | 10 | Periodic Tasks | 1 | MEDIUM | `periodic-` | | 11 | Serialization | 1 | MEDIUM | `serial-` | | 12 | Worker Management | 1 | MEDIUM | `worker-` | | 13 | Signals and Events | 1 | MEDIUM | `signals-` | | 14 | Remote Control and Inspection | 1 | MEDIUM | `remote-` | | 15 | Testing Celery Applications | 1 | MEDIUM-HIGH | `testing-` | | 16 | Daemonization and Process Management | 1 | MEDIUM | `daemonization-` | | 17 | Debugging Celery Applications | 1 | HIGH | `debugging-` | | 18 | Extensions and Bootsteps Customization | 1 | LOW | `extensions-` | | 19 | Advanced Patterns | 1 | LOW | `advanced-` | ## Quick Reference ### 1. Configuration & Setup (CRITICAL) - `config-broker-setup`: Properly configure message broker (Redis/RabbitMQ) - `config-environment-separation`: Separate configurations for different environments ### 2. Task Definition & Execution (CRITICAL) - `task-calling-methods`: Choose appropriate calling methods (delay/apply_async/call) - `task-atomic-operations`: Ensure tasks are atomic and idempotent ### 3. Error Handling & Reliability (HIGH) - `error-retry-strategy`: Implement intelligent retry mechanisms ### 2. Task Definition & Execution (CRITICAL) - `task-calling-methods`: Choose appropriate calling methods (delay/apply_async/call) - `task-atomic-operations`: Ensure tasks are atomic and idempotent - `task-options-configuration`: Configure task behavior and reliability options ### 3. Error Handling & Reliability (HIGH) - `error-retry-strategy`: Implement intelligent retry mechanisms ### 4. Canvas: Designing Work-flows (HIGH) - `canvas-chain-workflows`: Sequential task execution and error propagation - `canvas-group-parallel`: Parallel task execution for performance - `canvas-chord-synchronization`: Complex workflows with parallel execution and synchronization - `canvas-map-starmap`: Efficient batch processing and data transformations ### 5. Monitoring & Logging (HIGH) - `monitoring-task-tracking`: Track task execution and performance - `perf-monitoring-metrics`: Performance optimization and issue detection ### 6. Performance & Scaling (MEDIUM-HIGH) - `perf-concurrency-tuning`: Optimize worker concurrency settings ### 7. Security (MEDIUM-HIGH) - `security-task-authentication`: Prevent unauthorized task execution ### 8. Result Backends (MEDIUM-HIGH) - `result-backend-selection`: Choose appropriate result backend - `result-expiry-management`: Manage result expiration policies ### 9. Routing & Queues (MEDIUM) - `routing-task-distribution`: Intelligent task routing and queue management ### 10. Periodic Tasks (MEDIUM) - `periodic-celery-beat`: Reliable scheduled task execution ### 11. Serialization (MEDIUM) - `serial-data-handling`: Reliable data transmission and storage ### 12. Worker Management (MEDIUM) - `worker-lifecycle-management`: Reliable worker operation and resource management ### 13. Signals and Events (MEDIUM) - `signals-events-system`: Hook into Celery events for monitoring and extension ### 14. Remote Control and Inspection (MEDIUM) - `remote-control-inspection`: Runtime management and troubleshooting of workers ### 15. Testing Celery Applications (MEDIUM-HIGH) - `testing-celery-applications`: Comprehensive testing strategies for async applications ### 16. Daemonization and Process Management (MEDIUM) - `daemonization-process-management`: Production deployment and service management ### 17. Debugging Celery Applications (HIGH) - `debugging-celery-applications`: Effective troubleshooting and issue resolution ### 18. Advanced Performance Optimization (MEDIUM-HIGH) - `advanced-performance-optimization`: High-throughput optimization techniques ### 19. Extensions and Bootsteps Customization (LOW) - `extensions-bootsteps-customization`: Advanced customization and extension ### 20. Advanced Patterns (LOW) - `advanced-task-chaining`: Complex workflow orchestration ## How to Use Read individual rule files for detailed explanations and code examples: ``` rules/config-broker-setup.md # Broker configuration rules/task-calling-methods.md # Task execution methods rules/task-options-configuration.md # Task behavior options rules/canvas-chain-workflows.md # Sequential workflows rules/canvas-group-parallel.md # Parallel execution rules/canvas-chord-synchronization.md # Complex synchronization rules/canvas-map-starmap.md # Batch processing rules/monitoring-task-tracking.md # Task monitoring rules/result-backend-selection.md # Result storage rules/routing-task-distribution.md # Task routing rules/periodic-celery-beat.md # Scheduled tasks rules/serial-data-handling.md # Data serialization rules/signals-events-system.md # Event hooking rules/remote-control-inspection.md # Worker management rules/testing-celery-applications.md # Testing strategies rules/security-task-authentication.md # Task security rules/daemonization-process-management.md # Production deployment rules/debugging-celery-applications.md # Troubleshooting rules/advanced-performance-optimization.md # High-performance tuning rules/extensions-bootsteps-customization.md # Advanced customization ``` Each rule file contains: - Impact level and category - Problem description with consequences - Solution with code examples - Common mistakes to avoid - When to apply the rule ## Reference Documentation Detailed guides for complex topics: ``` references/worker-management.md # Worker deployment and scaling ``` *Additional reference guides planned: monitoring setup, troubleshooting, Canvas patterns* ## Full Compiled Document For the complete Celery best practices guide with all rules expanded: `AGENTS.md` --- *Celery Version Support: Compatible with Celery 4.x, 5.x, and 6.x (latest)* skills/celery-skill/SKILL.md