Apache Airflow AI Skill Guide
Overview & Engine Architecture
Airflow schedules DAGs of tasks executed by workers; the scheduler parses DAG files, the metadata DB stores run state, and executors (Local/Celery/Kubernetes) run task instances. Agents write idempotent tasks, set explicit retries/timeouts, avoid top-level heavy I/O in DAG files, and pass data via XCom sparingly (or external storage).
DAG file -> scheduler -> executor/workers
|
metadata DB (runs, XCom)
|
task logs / sensors
When to use this skill
- Time-based or data-aware batch pipelines
- Orchestrating dbt, Spark, warehouse SQL, ML batch jobs
- Backfills with clear logical dates
Operational directives
- Keep DAG top-level code fast (imports + structure only).
- Tasks must be idempotent for a given
data_interval / logical date.
- Set
retries, retry_delay, and execution_timeout intentionally.
- Prefer pushing large payloads to object storage over big XComs.
- Never commit connection passwords; use Airflow Connections / secrets backend.
Minimal DAG
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash import BashOperator
with DAG(
dag_id="orders_daily",
start_date=datetime(2026, 1, 1),
schedule="@daily",
catchup=False,
default_args={"retries": 2, "retry_delay": timedelta(minutes=5)},
tags=["orders"],
) as dag:
extract = BashOperator(
task_id="extract",
bash_command="python /opt/airflow/jobs/extract_orders.py --date {{ ds }}",
)
dbt_run = BashOperator(
task_id="dbt_run",
bash_command="cd /opt/dbt && dbt build --select marts.* --vars '{run_date: {{ ds }}}'",
)
extract >> dbt_run
Useful CLI
airflow dags list
airflow dags test orders_daily 2026-08-26
airflow tasks test orders_daily extract 2026-08-26
Common failures
| Symptom |
Cause |
Fix |
| DAG not appearing |
import error / parse fail |
check scheduler logs |
| Zombie / stuck tasks |
worker death |
timeouts; health checks |
| Huge backfill load |
catchup=True |
limit; clear carefully |
| Sensor hanging |
wrong poke / mode |
reschedule mode; timeouts |
Best practices
- One business pipeline per DAG id; stable task ids for clear history.
- Use datasets/data-aware scheduling when producers/consumers share tables.
- Pin provider package versions with Airflow constraints.
- Alert on SLA misses and failed task emails/Slack callbacks.
Limitations
- Not a streaming engine; pair with Kafka/Flink for continuous event processing.
- Executor/deployment topology (MWAA, Composer, K8s) changes ops details.
- This skill does not replace capacity planning for workers/metadata DB.
Related skills
@prefect - alternative Python-native orchestration@dbt - SQL models often invoked from Airflow@spark - heavy distributed tasks
1---2name: airflow3description: Operational skill for Apache Airflow: DAGs, operators, sensors, scheduling, retries, and production task hygiene.4---56# Apache Airflow AI Skill Guide78## Overview & Engine Architecture910Airflow schedules DAGs of tasks executed by workers; the scheduler parses DAG files, the metadata DB stores run state, and executors (Local/Celery/Kubernetes) run task instances. Agents write idempotent tasks, set explicit retries/timeouts, avoid top-level heavy I/O in DAG files, and pass data via XCom sparingly (or external storage).1112```13DAG file -> scheduler -> executor/workers14 |15 metadata DB (runs, XCom)16 |17 task logs / sensors18```1920## When to use this skill2122- Time-based or data-aware batch pipelines23- Orchestrating dbt, Spark, warehouse SQL, ML batch jobs24- Backfills with clear logical dates2526## Operational directives27281. Keep DAG top-level code fast (imports + structure only).292. Tasks must be idempotent for a given `data_interval` / logical date.303. Set `retries`, `retry_delay`, and `execution_timeout` intentionally.314. Prefer pushing large payloads to object storage over big XComs.325. Never commit connection passwords; use Airflow Connections / secrets backend.3334## Minimal DAG3536```python37from datetime import datetime, timedelta38from airflow import DAG39from airflow.operators.bash import BashOperator4041with DAG(42 dag_id="orders_daily",43 start_date=datetime(2026, 1, 1),44 schedule="@daily",45 catchup=False,46 default_args={"retries": 2, "retry_delay": timedelta(minutes=5)},47 tags=["orders"],48) as dag:49 extract = BashOperator(50 task_id="extract",51 bash_command="python /opt/airflow/jobs/extract_orders.py --date {{ ds }}",52 )53 dbt_run = BashOperator(54 task_id="dbt_run",55 bash_command="cd /opt/dbt && dbt build --select marts.* --vars '{run_date: {{ ds }}}'",56 )57 extract >> dbt_run58```5960## Useful CLI6162```bash63airflow dags list64airflow dags test orders_daily 2026-08-2665airflow tasks test orders_daily extract 2026-08-2666```6768## Common failures6970| Symptom | Cause | Fix |71| --- | --- | --- |72| DAG not appearing | import error / parse fail | check scheduler logs |73| Zombie / stuck tasks | worker death | timeouts; health checks |74| Huge backfill load | catchup=True | limit; clear carefully |75| Sensor hanging | wrong poke / mode | reschedule mode; timeouts |7677## Best practices7879- One business pipeline per DAG id; stable task ids for clear history.80- Use datasets/data-aware scheduling when producers/consumers share tables.81- Pin provider package versions with Airflow constraints.82- Alert on SLA misses and failed task emails/Slack callbacks.8384## Limitations8586- Not a streaming engine; pair with Kafka/Flink for continuous event processing.87- Executor/deployment topology (MWAA, Composer, K8s) changes ops details.88- This skill does not replace capacity planning for workers/metadata DB.8990## Related skills9192- `@prefect` - alternative Python-native orchestration93- `@dbt` - SQL models often invoked from Airflow94- `@spark` - heavy distributed tasks
alivirgo