# Airflow

> Operate a remote Apache Airflow 2.x or 3.x deployment (list/trigger/monitor DAGs, tasks, variables, connections, pools, backfills) and deploy DAG files to S3 or a dags folder via the `datus airflow` CLI

- Skill: `datus-ai/airflow` (Agent Skill)
- Install (CLI): `npx skillmds@latest add datus-ai/airflow`
- Raw SKILL.md: https://api.skillmd.com/api/skills/datus-ai/airflow/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: datus-ai (https://skillmd.com/u/datus-ai)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/datus-ai/airflow

---


# Airflow

`datus airflow` drives a remote Apache Airflow deployment through REST API v1
(Airflow 2.x, Basic Auth) or v2 (Airflow 3.x, JWT). Command groups mirror the
Airflow CLI. Global usage:

```
datus airflow [--profile <env>] <group> <subcommand> [args...]
```

`--profile` (before the group) selects the configured environment; add
`-o json` to any list/get command for full machine-readable output (tables
show a curated column subset). Destructive commands prompt for confirmation —
always pass `-y/--yes` when running non-interactively.

## Profile scope limits

An environment may restrict itself. The `## Airflow` prompt section shows this
per environment as `commands=...` and `dag_prefix=...`; check it before
composing a command rather than discovering the limit by failing.

- **`dag_prefix=<prefix>`** — every `dag_id` argument must start with it.
  Out-of-scope ids exit 2 *without contacting the server*, so a rejection tells
  you nothing about whether that DAG exists. `dags list`, `dags list-runs`
  (across all DAGs) and `assets events` silently drop rows outside the prefix
  and report the count on stderr. Because they carry no `dag_id` to check,
  `assets materialize` and `backfill pause|unpause|cancel` are unavailable —
  use `dags trigger <dag_id>` instead of materializing an asset.
- **`commands=<groups>`** — only those top-level groups exist. Anything else
  exits 2 with a policy error; it is not a typo, do not retry variations.

Not covered by `dag_prefix`: `variables`, `connections` and `pools` are
instance-wide objects in Airflow, so their list/get/export output is never
filtered — treat what you see there as shared with other teams. `dags
list-import-errors` is also unfiltered (errors are keyed by filename, which
does not map to a dag_id). `dags deploy` / `undeploy` target *file paths*, which
the prefix does not constrain — only the `--verify <dag_id>` argument is checked.

These limits are guardrails for you, not a security boundary; never work around
one by, say, deploying a file that defines an out-of-scope DAG. If a task
genuinely needs something outside the scope, say so and let the user widen the
profile.

## DAGs

```
datus airflow dags list [-p '%pattern%'] [-t TAG] [--paused|--unpaused] [-o json]
datus airflow dags details <dag_id>
datus airflow dags show <dag_id>              # ASCII task dependency tree
datus airflow dags source <dag_id>            # DAG file source code
datus airflow dags pause|unpause <dag_id>...
datus airflow dags trigger <dag_id> [-c '{"k":"v"}'] [-r RUN_ID] [-l 2026-01-01T00:00:00Z] [--note TEXT] [--wait]
datus airflow dags state <dag_id> <run_id>
datus airflow dags list-runs [<dag_id>] [--state failed] [--limit 20]
datus airflow dags clear-run <dag_id> <run_id> [--only-failed] [--dry-run] [-y]
datus airflow dags delete <dag_id> -y         # removes ALL metadata; confirm with the user first
datus airflow dags next-execution <dag_id>
datus airflow dags list-import-errors
```

`trigger --wait` polls until the run finishes: exit 0 = success, 1 = failed.
Triggering a **paused** DAG exits 2 before creating any run — the scheduler
would leave that run `queued` forever; `dags unpause <dag_id>` first (and ask
the user before unpausing something they did not mention).
Omit `<dag_id>` in `list-runs` to list runs across all DAGs.

## Deploying DAG files

```
datus airflow dags deploy <file-or-dir>... [--dest s3://bucket/dags/ | /path/to/dags]
    [--prefix team_a] [--prune -y] [--all-files] [--dry-run]
    [--verify <dag_id> [--verify-timeout 120]]
```

- `--dest` defaults to the profile's `dags_folder`. S3 targets work out of
  the box (boto3 ships with the plugin).
- Directories are scanned recursively for `*.py` / `*.zip`.
- `--verify <dag_id>` waits until the scheduler re-parsed that DAG and fails
  fast when the deployed file causes an import error — prefer it, it turns a
  blind upload into a checked deployment.
- `--prune` deletes target files not in this deployment — destructive, needs
  `-y`; use `--dry-run` first.

To delete individual files from the target (paths relative to the target
root, as printed by deploy):

```
datus airflow dags undeploy team_a/old_dag.py [--dest ...] [--dry-run] [-y]
```

The DAG goes stale on the next parse; follow with
`datus airflow dags delete <dag_id> -y` to also drop its metadata.

## Tasks

```
datus airflow tasks list <dag_id>
datus airflow tasks state <dag_id> <run_id> <task_id> [--map-index N]
datus airflow tasks states-for-dag-run <dag_id> <run_id>
datus airflow tasks logs <dag_id> <run_id> <task_id> [try_number] [--full-content]
datus airflow tasks clear <dag_id> [-t 'regex'] [-r RUN_ID] [--only-failed] [--dry-run] [-y]
datus airflow tasks failed-deps <dag_id> <run_id> <task_id>
```

Typical debugging flow: `dags list-runs <dag_id> --state failed` →
`tasks states-for-dag-run <dag_id> <run_id>` → `tasks logs <dag_id> <run_id>
<task_id>` → fix → `tasks clear <dag_id> -r <run_id> --only-failed -y`.

## Variables / Connections / Pools

```
datus airflow variables list|get KEY [-d DEFAULT]|set KEY VALUE [-j]|delete KEY|import FILE|export FILE
datus airflow connections list|get ID|add ID (--conn-uri URI | --conn-json '{...}' | --conn-type ...)|delete ID|test [ID]|import FILE|export FILE
datus airflow pools list|get NAME|set NAME SLOTS DESCRIPTION|delete NAME|import FILE|export FILE
```

Connection passwords are masked in output unless `--show-secrets`; exports
contain clear-text secrets — never paste an export back into chat.

## Assets, backfills, server info

```
datus airflow assets list|details --name N|materialize --name N|events [--asset-id N]
datus airflow backfill create --dag-id D --from-date ISO --to-date ISO [--dry-run] | list --dag-id D | pause|unpause|cancel ID
datus airflow version | health | providers list | plugins | config list | config get-value SECTION OPTION | jobs check
```

## Exit codes

0 success · 1 runtime/API error (also: failed run with `--wait`, failed
connection test, unhealthy `health`) · 2 usage error · 3 profile/config
error · 8 missing dependency (boto3, if the environment stripped it).

`assets` and the top-level `backfill` API are Airflow 3/API v2 features. The
Airflow 2/API v1 compatibility path covers DAG, run, task, log, variable,
connection, pool, server-info, and DAG deployment operations.