frank-luongt name: airflow-dag-patterns description: Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs. tags: [ai, data-engineering]
Apache Airflow DAG Patterns
Production-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.
Use this skill when
- Creating data pipeline orchestration with Airflow
- Designing DAG structures and dependencies
- Implementing custom operators and sensors
- Testing Airflow DAGs locally
- Setting up Airflow in production
- Debugging failed DAG runs
Do not use this skill when
- You only need a simple cron job or shell script
- Airflow is not part of the tooling stack
- The task is unrelated to workflow orchestration
Instructions
- Identify data sources, schedules, and dependencies.
- Design idempotent tasks with clear ownership and retries.
- Implement DAGs with observability and alerting hooks.
- Validate in staging and document operational runbooks.
Safety
- Avoid changing production DAG schedules without approval.
- Test backfills and retries carefully to prevent data duplication.