Sync Grapher Schema

Sync Grapher Schema

The grapher chart-config schema is owned by the web team in owid-grapher and published at https://files.ourworldindata.org/schemas/grapher-schema.NNN.json. It is mutated in place without version bumps (e.g. dumbbell plots landed in .010 directly), so when it changes upstream, four things in this repo need to follow:

Unit tests enforce consistency between all of these (tests/test_schema_types_generation.py, test_grapher_config_schema_sync in tests/test_metadata_schemas.py), so partial syncs fail CI. Full background: docs/guides/grapher-schema-sync.md.

Entry points

A. Completing a bot PR (the common case). The scheduled workflow (.github/workflows/sync-grapher-schema.yml) detected an upstream change and opened a draft PR on the auto-sync-grapher-schema branch with the automatic part (refreshed vendored copy + regenerated types) already committed.

• Check out that branch — do NOT create a new PR (skip step 0; step 1's refresh is already done, just read the committed vendored diff).
• The PR's failing test_grapher_config_schema_sync output is the todo list — usually just steps 2-3 below.
• ⚠️ If upstream changes again before this PR merges, the workflow force-updates the branch and clobbers manual commits. Finish promptly; if the sync needs longer, move the work to your own branch (git checkout -b <new> + close the bot PR).
• When done: push, mark the PR ready for review.

B. Ad-hoc / from scratch. Someone announced a change and you're not waiting for the cron (alternatively, trigger the workflow manually: gh workflow run sync-grapher-schema.yml). Follow all steps below.

C. Version bump. The workflow opened a "New grapher schema version published upstream" issue → see the "Version bump" section at the bottom.

Workflow

0. Branch + PR

(Entry point B only.) Use the standard flow: .venv/bin/etl pr "sync grapher schema (<short summary>)" chore, unless the user wants the changes on the current branch.

1. Refresh the vendored schema

(Entry point A: already committed by the workflow — just read the diff with git show on the bot commit, then continue at step 2.)

.venv/bin/python scripts/generate_schema_types.py --refresh
git diff schemas/grapher-schema.*.json

• Diff is empty → nothing changed upstream at the pinned version. Check whether a new schema version was published (see "Version bump" below); otherwise report there's nothing to sync and stop.
• Diff is non-empty → read it carefully. It is the authoritative list of what must propagate in steps 2-3. Summarize it for the user (new properties, new enum values, changed descriptions/defaults).

2. Propagate to schemas/multidim-schema.json

Only needed for new top-level properties (new chart-type config objects like dumbbell, new view-level fields). Existing $refs resolve against the live schema automatically.

For each new upstream property that makes sense in a multidim/explorer view, add a $ref entry to the view config properties block (search for "chartTypes" to find it). The same applies to schemas/explorer-schema.json. Refs are local relative refs to the vendored copy (resolved offline by Collection.validate_schema):

"<newProp>": {
    "$ref": "grapher-schema.NNN.json#/properties/<newProp>"
},

Lesson learned (#6196 → #6200): forgetting this step is how dumbbell went missing — the generated types were patched by hand instead, which regeneration would have destroyed. Never edit schema_types.py directly.

3. Propagate to schemas/dataset-schema.json

The grapher config is embedded inline (not $ref'd) under ...variables.additionalProperties.properties.presentation.properties.grapher_config.properties. Mirror every change from the step-1 diff into that block — new properties, new enum values, updated descriptions.

Preserve these deliberate ETL-side deviations (do NOT "fix" them to match upstream):

• Extra properties not in upstream: data, includedEntities.
• chartTypes enum includes WorldMap (not upstream).
• Many enum fields are wrapped in oneOf with a Jinja escape hatch — keep the wrapper, edit only the enum branch:

  "oneOf": [
      { "enum": [...sync these values...] },
      { "type": "string", "pattern": "{definitions" }
  ]
  

  (Some metadata fields use "pattern": "<%" instead — same idea.)

4. Regenerate the Python types

.venv/bin/python scripts/generate_schema_types.py
git diff etl/collection/model/schema_types.py

Sanity-check the diff: it should reflect exactly the upstream changes (plus any multidim $ref additions). If a class or field unexpectedly disappears, a $ref is probably missing (step 2).

Hand-written types (e.g. GroupViewsConfig) live in etl/collection/model/params.py — never add them to the generated file.

5. Validate

.venv/bin/pytest tests/test_schema_types_generation.py tests/test_metadata_schemas.py tests -k "collection or schema" -m "not integration" -q
make check

test_grapher_config_schema_sync pinpoints any enum value or property still missing from the embedded block (exact JSON path in the failure message) — iterate on step 3 until green.

6. Commit & PR description

Commit with ✨🤖. In the PR body, list the upstream changes synced (link the Slack announcement if there is one) and which of the four files each change touched. For entry point A, mark the bot PR ready for review instead of writing a new body — just add a comment summarizing the manual propagation you did.

Version bump (upstream publishes grapher-schema.NNN+1)

Rarer case — when the web team publishes a new schema version instead of mutating in place. Detected by the integration test test_no_newer_grapher_schema_version (compares the $id of upstream grapher-schema.latest.json against DEFAULT_GRAPHER_SCHEMA).

.venv/bin/python scripts/generate_schema_types.py --bump-version

That one command reads the new version from grapher-schema.latest.json's $id, vendors it as schemas/grapher-schema.MMM.json, deletes the old copy, repoints every $ref in schemas/multidim-schema.json + schemas/explorer-schema.json, and regenerates schema_types.py. Nothing in etl/config.py is hand-edited: DEFAULT_GRAPHER_SCHEMA is derived from whichever schema is vendored (vendored_grapher_schema_id()), so it follows automatically — and it always resolves to a concrete version, never latest, since grapher's config migrations are keyed on the version.

Then, by hand:

1. Review the upstream diff: git show HEAD:schemas/grapher-schema.NNN.json | diff -u - schemas/grapher-schema.MMM.json.
2. Continue from step 3 above — mirror new/changed properties into the embedded grapher_config block in schemas/dataset-schema.json, preserving the deliberate ETL-side deviations. test_grapher_config_schema_sync fails until this is done.
3. Check whether the command reported leftover mentions of the old filename outside $refs (the grapher_schema examples in multidim-schema.json, which show what a new config should pin).

--bump-version is idempotent — it prints "already vendoring the newest published schema" and changes nothing when there is no new version, so it is safe to run blind.

Don't bump the grapher_schema pins in MDIM configs

Every multidim and single-chart config pins grapher_schema: "NNN" — required, with no fallback (required in schemas/multidim-schema.json, re-checked by Collection.validate_grapher_schema_pinned(), and swept offline by test_multidim_configs_pin_grapher_schema). Leave those pins at their old version. They record what each config was authored against, which is what lets grapher migrate them to MMM on upsert. Bumping them would tell grapher the configs are already current and skip the migration — the exact failure the pins exist to prevent.

The one thing to check: --bump-version repoints multidim view-config validation at the new version, so a config that is no longer valid under MMM will now fail Collection.validate_schema(). Fix the config and bump only that config's pin, since at that point it genuinely was re-authored against MMM.

Views can also carry their own $schema inside a config block, which overrides the collection-level pin (grapher spreads the view config last). As of #6705 follow-up no step does this any more, and ETL warns if one reappears — so treat a hit from grep -rn '\$schema' etl/steps/viz/chart as something to remove rather than to bump.

One caveat on "leave the pins alone": that holds for pins that are true. A pin that contradicts its own config body — pinned 005 while the config uses chartTypes, which only exists from 006 (the 005→006 migration creates it) — is stale, not a record, and leaving it makes grapher run migrations over a config they were never meant to touch. Check a suspicious pin against the properties of that schema version (curl https://files.ourworldindata.org/schemas/grapher-schema.NNN.json) and correct it to the version the config is actually written against.