# Create Etl Steps > Create vanilla meadow, garden, and grapher ETL step files by invoking the wizard's cookiecutter templates, given a snapshot path. - Skill: `owid/create-etl-steps` (Agent Skill) - Install (CLI): `npx skillmds@latest add owid/create-etl-steps` - Raw SKILL.md: https://api.skillmd.com/api/skills/owid/create-etl-steps/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Data & Analytics - Author: owid (https://skillmd.com/u/owid) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/owid/create-etl-steps --- # Create ETL Steps Create meadow, garden, and grapher step files for a given snapshot, by running the same cookiecutter templates the wizard runs. > **Never copy the templates into this file.** `apps/wizard/etl_steps/cookiecutter/{meadow,garden,grapher}/` is the single source of truth, and this skill invokes it via `generate_step_to_channel`. An earlier version of this skill embedded hand-copied templates; they drifted (the multi-snapshot meadow branch and `non_redistributable` for private datasets both went missing) while the copies that hadn't drifted made the rest look current. If a template needs changing, change it in `apps/wizard/etl_steps/cookiecutter/`. `/create-dataset` calls this skill at its Step 5. Keep the two consistent: if the inputs or generated files change here, check whether `create-dataset/SKILL.md` needs a matching edit, and make it in the same commit. ## Inputs Required: - `snapshot_path` — in the format `namespace/version/short_name` (e.g. `washu/2026-04-22/pm25_air_pollution`) Optional: - `dag_file` — which DAG file to add entries to (e.g. `environment`, `climate`). If not provided, ask the user. - `is_private` — default `False`. Affects both the generated metadata and the DAG URIs (see steps 4 and 5). - `update_period_days` — default `365`. - `topic_tags` — default none. ## Workflow ### 1. Parse the snapshot path Extract: - `namespace` — e.g. `washu` - `version` — e.g. `2026-04-22` - `short_name` — e.g. `pm25_air_pollution` ### 2. Find the snapshot file extension Look in `snapshots///` for a `.dvc` file matching `.*`. The part between `.` and `.dvc` is the `file_extension`. For example: `pm25_air_pollution.csv.dvc` → `file_extension = csv` The full snapshot filename is `.`. Collect **all** the snapshot filenames the meadow step should read — if the chain has more than one snapshot, pass them all in step 4 and the meadow template generates the multi-snapshot loop by itself. ### 3. Determine the DAG file If the user has not specified a DAG file, list the available files in `dag/` (excluding `archive/`) and ask the user which one to use. ### 4. Generate the step files Call the wizard's generator once per channel. It creates the directories, renders the templates, runs ruff on the generated Python, and copies the result into `etl/steps/data//`: ```bash .venv/bin/python -c " from apps.utils.files import generate_step_to_channel from apps.wizard.etl_steps.utils import COOKIE_STEPS, remove_playground_notebook from etl.owners import resolve_owner import subprocess namespace, version, short_name = '', '', '' snapshot_names = ['.'] # every snapshot the meadow step reads dag_file, is_private = '.yml', False update_period_days, topic_tags = 365, [] git_name = subprocess.check_output(['git', 'config', 'user.name'], text=True).strip() owner = resolve_owner(git_name) # The garden .meta.yml only renders an `owners:` block when `owner` is truthy, so an # unresolved name would silently produce metadata with no accountable owner. assert owner, f'resolve_owner() did not recognize git user.name={git_name!r} — ask for a canonical owner' common = { 'namespace': namespace, 'short_name': short_name, 'version': version, 'add_to_dag': True, 'dag_file': dag_file, 'is_private': is_private, } per_channel = { 'meadow': {'channel': 'meadow', 'snapshot_names_with_extension': snapshot_names}, # topic_tags must be a pre-joined string, not a list: cookiecutter renders a list # as only its first element. 'garden': { 'channel': 'garden', 'meadow_version': version, 'update_period_days': update_period_days, 'topic_tags': ('- ' + '\n- '.join(topic_tags)) if topic_tags else '', 'owner': owner, }, 'grapher': {'channel': 'grapher', 'garden_version': version}, } for channel, extra in per_channel.items(): dataset_dir = generate_step_to_channel(cookiecutter_path=COOKIE_STEPS[channel], data={**common, **extra}) # The meadow and garden cookiecutters both ship a playground.ipynb. The wizard keeps it only # for garden when the user asked for a notebook; this skill never does, so always drop it. remove_playground_notebook(dataset_dir) print(f'{channel}: {dataset_dir}') " ``` Notes on this call: - **Run the channels one at a time, never concurrently.** `generate_step` writes a temporary `cookiecutter.json` into the template directory and deletes it afterwards, so two simultaneous runs corrupt each other's context. - **Every variable a template references must be present in `data`.** There is no committed `cookiecutter.json` supplying defaults, so a missing key is a Jinja `UndefinedError`, not a silent blank. The dicts above are what `apps/wizard/etl_steps/forms.py:309` passes; if a template gains a variable, it has to be added here too. - **If the `owner` assert fires, stop and ask** which colleague is the accountable owner, then set `owner` to that canonical name and re-run. `resolve_owner` only recognizes the git identities in `etl/owners.py`, so it returns `None` on an unmapped `git config user.name` — a cloud sandbox, a fresh checkout, or a name spelled differently from the enum. The wizard form degrades to an empty string there, but it has a human in front of it who can see the missing block; a skill run does not, and CLAUDE.md requires `owners` on every dataset. Pick the name from the `schemas/dataset-schema.json` enum, and add the git identity to `etl/owners.py` if it is a colleague who is simply missing from the map. - `generate_step` prints the context dictionary to stdout, and importing `apps.wizard` logs a `No runtime found, using MemoryCacheStorageManager` warning from Streamlit. Both are expected noise, not errors. - Use `/create-playground` if the user does want a playground notebook, rather than keeping the cookiecutter's copy. Files generated, after the playground removal: meadow `.py`; garden `.py`, `.meta.yml`, `.countries.json`, `.excluded_countries.json`; grapher `.py`. Verify the notebook is gone — leaving one behind is the easiest thing to get wrong here, since two of the three channels ship it. ### 5. Add DAG entries Append the following entries to `dag/.yml` under the `steps:` key, using `ruamel_load` / `ruamel_dump` to preserve comments. For a private dataset every `data://` below becomes `data-private://` (matching `private_suffix` in the wizard form): ```yaml data://meadow///: - snapshot:////. data://garden///: - data://meadow/// data://grapher///: - data://garden/// ``` **The snapshot URI has its own prefix, driven by the snapshot's `.dvc`, not by `is_private`.** A snapshot whose `.dvc` sets `is_public: false` is referenced as `snapshot-private://`; everything else as `snapshot://`. Read `is_public` out of each `.dvc` rather than assuming — a private dataset is normally built on private snapshots, but the two flags are independent, and a public snapshot can feed a private dataset. Getting this wrong is silent: `snapshot-private://` builds a `SnapshotStepPrivate`, whose `run()` asserts `is_public is False` before pulling, and `--private` filtering keys off the prefix too, so a private snapshot mislabeled `snapshot://` loses that assert and is no longer excluded from a public run. Every one of the 294 private snapshots in the active DAG uses `snapshot-private://`, with no exceptions — a plain `snapshot://` on a private snapshot would be the first. List every snapshot from step 2 as a dependency of the meadow step, not just the first. ```python from etl.files import ruamel_load, ruamel_dump from etl.snapshot import Snapshot base = "//" snapshot_names = ["."] # every snapshot from step 2 is_private = False data_prefix = "data-private" if is_private else "data" # Each snapshot's own is_public decides its prefix, independently of is_private. snapshot_uris = [] for name in snapshot_names: path = f"//{name}" prefix = "snapshot" if Snapshot(path).metadata.is_public else "snapshot-private" snapshot_uris.append(f"{prefix}://{path}") dag_path = "dag/.yml" with open(dag_path, "r") as f: data = ruamel_load(f) data["steps"][f"{data_prefix}://meadow/{base}"] = snapshot_uris data["steps"][f"{data_prefix}://garden/{base}"] = [f"{data_prefix}://meadow/{base}"] data["steps"][f"{data_prefix}://grapher/{base}"] = [f"{data_prefix}://garden/{base}"] with open(dag_path, "w") as f: f.write(ruamel_dump(data)) ``` ### 6. Name the checks the filled-in steps will need **Don't run `/check-outdated-practices` here.** What this skill produces is untouched cookiecutter output, and the templates are verified clean against the detector's full pattern set — so running it on the scaffold is a guaranteed no-op. The patterns it looks for enter when the scaffold is *adapted*: real load logic, harmonization, aggregations, hand-copied helper modules like `*_omms.py`. That's why `/create-dataset` runs it at its Step 5, after adapting these files, and `/update-dataset` runs it at step 1b on files `etl update` carried over from the previous version — neither of which is scaffold output. Template drift is covered by the detector itself rather than by a check here: `apps/wizard/**/cookiecutter/**` is in the scope of every pattern, so a stale practice in a template shows up in the editor as soon as someone opens it. So report the checks the person will need once the steps do something, rather than running them on empty files. The metadata checks in particular have nothing to bite on yet — the scaffolded `.meta.yml` is entirely commented out: - `/check-outdated-practices` — **after** adapting the step `.py` files, and on any helper module copied in by hand - `/check-metadata-style` — user-facing text against the Writing and Style Guide - `/check-metadata-typos` — spelling - `/check-metadata-spacing` — Jinja rendering artifacts, once the metadata uses templates Also flag `.claude/rules/sanity-checks.md` if the garden step will do more than load-and-format: assertions are expected in the step, and the scaffold has none. ### 7. Report to the user List all files created and the DAG entries added, and the deferred checks from step 6 — saying plainly that nothing has been checked yet because there is nothing to check, so the next person doesn't read silence as a clean bill of health. Suggest running: ```bash .venv/bin/etlr // ```