# Airflow Setup

> Configure an environment profile for the `datus airflow` plugin (API endpoint, credentials, optional S3/dags-folder deployment target)

- Skill: `datus-ai/airflow-setup` (Agent Skill)
- Install (CLI): `npx skillmds@latest add datus-ai/airflow-setup`
- Raw SKILL.md: https://api.skillmd.com/api/skills/datus-ai/airflow-setup/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: datus-ai (https://skillmd.com/u/datus-ai)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/datus-ai/airflow-setup

---


# Airflow Setup

Use this skill when `datus airflow` is installed but has no configured
environment, or when the user wants to add another environment.

## Config structure

Profiles live under `agent.plugins.airflow.<profile>` in the config file named
by the `## Plugins` section of the system prompt:

```yaml
agent:
  plugins:
    airflow:
      prod:
        default: true                        # mark exactly one profile as default
        api_base_url: https://airflow.example.com/api/v1  # v1 suffix selects Airflow 2
        api_version: auto                           # auto | v1 | v2

        # auth — EITHER a static JWT token:
        token: ${AIRFLOW_API_TOKEN}          # secret — env var reference, never a literal
        # OR username + password (exchanged for a JWT at POST /auth/token):
        username: admin
        password: ${AIRFLOW_PASSWORD}        # secret — env var reference, never a literal

        # optional:
        verify_ssl: true                     # false or a CA bundle path for self-signed TLS
        timeout: 30                          # request timeout in seconds
        dags_folder: s3://my-bucket/dags/    # default target for `dags deploy`
                                             # (or a mounted path like /opt/airflow/dags)

        # optional scope guardrails (see "Scoping a profile" below):
        dag_id_prefix: team_a_               # only team_a_* DAGs; comma-separate several
        allow_commands: dags,tasks,version   # only these top-level groups

        s3:                                  # only for s3:// dags_folder, all optional
          region: us-east-1
          profile: my-aws-profile            # named AWS profile
          endpoint_url: http://minio:9000    # for MinIO/custom endpoints
          access_key_id: ${AWS_ACCESS_KEY_ID}         # secret — env var reference
          secret_access_key: ${AWS_SECRET_ACCESS_KEY} # secret — env var reference
          role_arn: arn:aws:iam::123456789012:role/dags-deployer  # assume this IAM role
          role_session_name: datus-deploy    # optional, default datus-airflow-plugin
          external_id: team-a                # optional, if the role trust policy requires it
```

## Steps

1. Ask the user for:
   - `api_base_url` — the Airflow web server root. An `/api/v1` suffix selects
     Airflow 2 with Basic Auth; `/api/v2` selects Airflow 3 with JWT. Without a
     suffix, `api_version` defaults to v2.
   - Auth method: a ready-made API token, **or** username + password. For the
     secret, have the user export an environment variable (e.g.
     `export AIRFLOW_PASSWORD=...`) and write `${VAR}` into the YAML — never a
     literal secret.
   - Whether they deploy DAGs through this plugin; if yes, the `dags_folder`
     target (`s3://bucket/prefix/` or a local/mounted path) and any S3
     specifics (region, named profile, custom endpoint). boto3 ships with the
     plugin, so S3 deployment works out of the box.
2. Write the profile into the config file named in the `## Plugins` preamble;
   mark the first profile `default: true`.
3. Verify with a cheap read-only call: `datus airflow version` (checks
   connectivity + auth), then `datus airflow dags list --limit 5`.
4. If deployment is configured, optionally verify with
   `datus airflow dags deploy <some-dag>.py --dry-run`.

If this environment cannot edit the config file (API / web deployment), tell
the user to edit `agent.yml` on the server instead.

## Scoping a profile

Offer these when one environment should only serve one team or project:

- `dag_id_prefix` — commands taking a `dag_id` refuse ids outside the prefix
  before any request; DAG listings are filtered to it. `assets materialize` and
  `backfill pause|unpause|cancel` become unavailable (no `dag_id` to check).
- `allow_commands` — allowlist of top-level groups, e.g. `dags,tasks,version`.
  Group level only; `dags list` is a config error, write `dags`.

Both appear in the system prompt per environment, so the agent knows the
boundary without probing for it.

Be explicit with the user that this is a **guardrail against mistakes, not a
security boundary** — anyone who can edit `agent.yml` or reach the Airflow API
bypasses it. Real multi-tenancy needs server-side enforcement (DAG-level RBAC
via FabAuthManager, or Airflow 3.2+ `[core] multi_team`), plus a separate
Airflow user per profile so the server also limits what the token can do.
Note that `variables`, `connections` and `pools` are instance-wide and cannot
be prefix-scoped at all — leave them out of `allow_commands` if that matters.

## Troubleshooting

- `login failed at .../auth/token` — username/password wrong, or the server's
  auth manager does not expose `POST /auth/token` (set `auth_token_url` if it
  lives elsewhere).
- `TLS verification failed` — set `verify_ssl` to the CA bundle path, or
  `false` as a last resort.
- 403 on `config` commands — server needs `AIRFLOW__API__EXPOSE_CONFIG=True`.
- 403/error on `connections test` — server needs
  `AIRFLOW__CORE__TEST_CONNECTION=Enabled`.