# Sync Grapher Schema

> Sync upstream grapher schema changes (new chart types, config fields, enum values) into the ETL repo — vendored schema, multidim-schema, dataset-schema, and regenerated Python types. Use when the scheduled sync workflow opened a draft PR or issue that needs completing, when the web team announces a grapher schema change ("new chart type in Grapher", "I added a field to the grapher config"), when someone asks to "sync the grapher schema", or when grapher configs fail ETL validation on fields that work fine in the grapher admin.

- Skill: `owid/sync-grapher-schema` (Agent Skill)
- Install (CLI): `npx skillmds@latest add owid/sync-grapher-schema`
- Raw SKILL.md: https://api.skillmd.com/api/skills/owid/sync-grapher-schema/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Data & Analytics
- Author: owid (https://skillmd.com/u/owid)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/owid/sync-grapher-schema

---


# Sync Grapher Schema

The grapher chart-config schema is owned by the web team in [`owid-grapher`](https://github.com/owid/owid-grapher/tree/master/packages/%40ourworldindata/grapher/src/schema) and published at `https://files.ourworldindata.org/schemas/grapher-schema.NNN.json`. **It is mutated in place without version bumps** (e.g. dumbbell plots landed in `.010` directly), so when it changes upstream, four things in this repo need to follow:

| File | Role | Sync mechanism |
|---|---|---|
| `schemas/grapher-schema.NNN.json` | Vendored copy of upstream, and the single source of truth for the version (`DEFAULT_GRAPHER_SCHEMA` is derived from its `$id`) | automatic (`--refresh`; `--bump-version` for a new version) |
| `schemas/multidim-schema.json` + `schemas/explorer-schema.json` | View config `$ref`s into the grapher schema | manual: add `$ref` for **new** properties |
| `schemas/dataset-schema.json` | Embedded `grapher_config` block (validates garden `.meta.yml`) | manual: mirror changes, preserve deviations |
| `etl/collection/model/schema_types.py` | Generated Python TypedDicts | automatic (regenerate) |

Unit tests enforce consistency between all of these (`tests/test_schema_types_generation.py`, `test_grapher_config_schema_sync` in `tests/test_metadata_schemas.py`), so partial syncs fail CI. Full background: `docs/guides/grapher-schema-sync.md`.

## Entry points

**A. Completing a bot PR** (the common case). The scheduled workflow (`.github/workflows/sync-grapher-schema.yml`) detected an upstream change and opened a draft PR on the `auto-sync-grapher-schema` branch with the automatic part (refreshed vendored copy + regenerated types) already committed.

- Check out that branch — do NOT create a new PR (skip step 0; step 1's refresh is already done, just read the committed vendored diff).
- The PR's failing `test_grapher_config_schema_sync` output is the todo list — usually just steps 2-3 below.
- ⚠️ If upstream changes again before this PR merges, the workflow **force-updates the branch and clobbers manual commits**. Finish promptly; if the sync needs longer, move the work to your own branch (`git checkout -b <new>` + close the bot PR).
- When done: push, mark the PR ready for review.

**B. Ad-hoc / from scratch.** Someone announced a change and you're not waiting for the cron (alternatively, trigger the workflow manually: `gh workflow run sync-grapher-schema.yml`). Follow all steps below.

**C. Version bump.** The workflow opened a "New grapher schema version published upstream" issue → see the "Version bump" section at the bottom.

## Workflow

### 0. Branch + PR

(Entry point B only.) Use the standard flow: `.venv/bin/etl pr "sync grapher schema (<short summary>)" chore`, unless the user wants the changes on the current branch.

### 1. Refresh the vendored schema

(Entry point A: already committed by the workflow — just read the diff with `git show` on the bot commit, then continue at step 2.)

```bash
.venv/bin/python scripts/generate_schema_types.py --refresh
git diff schemas/grapher-schema.*.json
```

- **Diff is empty** → nothing changed upstream at the pinned version. Check whether a *new schema version* was published (see "Version bump" below); otherwise report there's nothing to sync and stop.
- **Diff is non-empty** → read it carefully. It is the authoritative list of what must propagate in steps 2-3. Summarize it for the user (new properties, new enum values, changed descriptions/defaults).

### 2. Propagate to `schemas/multidim-schema.json`

Only needed for **new top-level properties** (new chart-type config objects like `dumbbell`, new view-level fields). Existing `$ref`s resolve against the live schema automatically.

For each new upstream property that makes sense in a multidim/explorer view, add a `$ref` entry to the view config properties block (search for `"chartTypes"` to find it). The same applies to `schemas/explorer-schema.json`. Refs are **local relative refs** to the vendored copy (resolved offline by `Collection.validate_schema`):

```json
"<newProp>": {
    "$ref": "grapher-schema.NNN.json#/properties/<newProp>"
},
```

Lesson learned (#6196 → #6200): forgetting this step is how `dumbbell` went missing — the generated types were patched by hand instead, which regeneration would have destroyed. Never edit `schema_types.py` directly.

### 3. Propagate to `schemas/dataset-schema.json`

The grapher config is **embedded inline** (not `$ref`'d) under `...variables.additionalProperties.properties.presentation.properties.grapher_config.properties`. Mirror every change from the step-1 diff into that block — new properties, new enum values, updated descriptions.

**Preserve these deliberate ETL-side deviations** (do NOT "fix" them to match upstream):

- Extra properties not in upstream: `data`, `includedEntities`.
- `chartTypes` enum includes `WorldMap` (not upstream).
- Many enum fields are wrapped in `oneOf` with a Jinja escape hatch — keep the wrapper, edit only the enum branch:
  ```json
  "oneOf": [
      { "enum": [...sync these values...] },
      { "type": "string", "pattern": "{definitions" }
  ]
  ```
  (Some metadata fields use `"pattern": "<%"` instead — same idea.)

### 4. Regenerate the Python types

```bash
.venv/bin/python scripts/generate_schema_types.py
git diff etl/collection/model/schema_types.py
```

Sanity-check the diff: it should reflect exactly the upstream changes (plus any multidim `$ref` additions). If a class or field unexpectedly *disappears*, a `$ref` is probably missing (step 2).

Hand-written types (e.g. `GroupViewsConfig`) live in `etl/collection/model/params.py` — never add them to the generated file.

### 5. Validate

```bash
.venv/bin/pytest tests/test_schema_types_generation.py tests/test_metadata_schemas.py tests -k "collection or schema" -m "not integration" -q
make check
```

`test_grapher_config_schema_sync` pinpoints any enum value or property still missing from the embedded block (exact JSON path in the failure message) — iterate on step 3 until green.

### 6. Commit & PR description

Commit with `✨🤖`. In the PR body, list the upstream changes synced (link the Slack announcement if there is one) and which of the four files each change touched. For entry point A, mark the bot PR ready for review instead of writing a new body — just add a comment summarizing the manual propagation you did.

## Version bump (upstream publishes grapher-schema.NNN+1)

Rarer case — when the web team publishes a new schema version instead of mutating in place. Detected by the integration test `test_no_newer_grapher_schema_version` (compares the `$id` of upstream `grapher-schema.latest.json` against `DEFAULT_GRAPHER_SCHEMA`).

```bash
.venv/bin/python scripts/generate_schema_types.py --bump-version
```

That one command reads the new version from `grapher-schema.latest.json`'s `$id`, vendors it as `schemas/grapher-schema.MMM.json`, deletes the old copy, repoints every `$ref` in `schemas/multidim-schema.json` + `schemas/explorer-schema.json`, and regenerates `schema_types.py`. **Nothing in `etl/config.py` is hand-edited**: `DEFAULT_GRAPHER_SCHEMA` is derived from whichever schema is vendored (`vendored_grapher_schema_id()`), so it follows automatically — and it always resolves to a concrete version, never `latest`, since grapher's config migrations are keyed on the version.

Then, by hand:

1. Review the upstream diff: `git show HEAD:schemas/grapher-schema.NNN.json | diff -u - schemas/grapher-schema.MMM.json`.
2. Continue from step 3 above — mirror new/changed properties into the embedded `grapher_config` block in `schemas/dataset-schema.json`, preserving the deliberate ETL-side deviations. `test_grapher_config_schema_sync` fails until this is done.
3. Check whether the command reported leftover mentions of the old filename outside `$ref`s (the `grapher_schema` `examples` in `multidim-schema.json`, which show what a *new* config should pin).

`--bump-version` is idempotent — it prints "already vendoring the newest published schema" and changes nothing when there is no new version, so it is safe to run blind.

### Don't bump the `grapher_schema` pins in MDIM configs

Every multidim and single-chart config pins `grapher_schema: "NNN"` — required, with no fallback (`required` in `schemas/multidim-schema.json`, re-checked by `Collection.validate_grapher_schema_pinned()`, and swept offline by `test_multidim_configs_pin_grapher_schema`). **Leave those pins at their old version.** They record what each config was authored against, which is what lets grapher migrate them to `MMM` on upsert. Bumping them would tell grapher the configs are already current and skip the migration — the exact failure the pins exist to prevent.

The one thing to check: `--bump-version` repoints multidim view-config validation at the new version, so a config that is no longer valid under `MMM` will now fail `Collection.validate_schema()`. Fix the config *and* bump only that config's pin, since at that point it genuinely was re-authored against `MMM`.

Views can also carry their own `$schema` inside a `config` block, which **overrides** the collection-level pin (grapher spreads the view config last). As of #6705 follow-up no step does this any more, and ETL warns if one reappears — so treat a hit from `grep -rn '\$schema' etl/steps/viz/chart` as something to remove rather than to bump.

One caveat on "leave the pins alone": that holds for pins that are *true*. A pin that contradicts its own config body — pinned `005` while the config uses `chartTypes`, which only exists from `006` (the 005→006 migration creates it) — is stale, not a record, and leaving it makes grapher run migrations over a config they were never meant to touch. Check a suspicious pin against the properties of that schema version (`curl https://files.ourworldindata.org/schemas/grapher-schema.NNN.json`) and correct it to the version the config is actually written against.