Create Explorer

Creating an Explorer

Explorers are OWID's multi-dimensional dashboards (e.g. ourworldindata.org/explorers/food-prices). They're authored as YAML in this repo and published by ETL at viz://explorer/<ns>/latest/<short>.

This skill is the explorer-flavored sibling of /create-multidim. They use the same paths.create_collection(...) engine and the same YAML schema for dimensions / views / definitions.common_views. The differences are:

If you're modifying an existing explorer (adjusting chart text, swapping a catalogPath, adding a dimension choice, reordering views), most of the deeper sections below don't apply — find the existing <short>.config.yml, edit, run etlr, done. The full structure is documented below for new explorers and substantial reshapes.

When to use this skill

• After a /migrate-explorer-csv, /migrate-explorer-grapher, or /migrate-explorer-indicator-legacy skill has produced (or already located) the upstream snapshot/meadow/garden/grapher chain, and now needs the explorer step.
• For a brand-new explorer where the data is already in ETL (skip directly to step 1).
• When porting an existing explorer's view layout (e.g. full-YAML → table-driven, or moving FAUST text from per-view YAML up into indicator metadata).

Step 1 — Files & directories

Every explorer is exactly two files plus a DAG entry:

etl/steps/viz/explorer/<ns>/latest/
├── <short>.py              # Python uses snake_case
└── <short>.config.yml

mkdir -p etl/steps/viz/explorer/<ns>/latest

Hyphens vs underscores (recurring source of confusion):

• The Python file path uses underscores: food_footprints.py, crop_yields.py.
• The explorer slug used in the URL and the short_name= argument keeps hyphens: food-footprints, crop-yields.
• paths.create_collection(short_name="<short-with-hyphens>") — pass the hyphenated slug.

Step 2 — Pick a construction style

The two ends of the spectrum, plus everything in between:

It's a spectrum, not a switch. Mix freely: use table-driven for the bulk of views, hand-list a handful of bespoke ones; or stay full-YAML but still push title/subtitle for the single-indicator views into garden metadata to remove duplication.

Strong fit for table-driven:

• Single-indicator views dominate. Each view is a thin wrapper around one indicator → that indicator's metadata is the right home for chart text. Avoids duplication between explorer YAML and the equivalent standalone chart, and keeps both in sync forever.
• Many views (>20) following the cartesian product of a few dimensions. Hand-listing them is repetitive; auto-expansion plus YAML dimensions is significantly less code.
• Upstream is a dimensional table (one row per country/year × dim_a × dim_b × …) with one indicator. create_collection(tb=tb, indicator_names=..., dimensions=...) matches this shape directly — model: migration/latest/migration_flows.py.
• Same indicators back standalone grapher charts. Pushing FAUST upstream means explorer view and standalone chart inherit the same text — no drift over time.

Stick with full-YAML when:

• Few views (<10), all bespoke (different chart types / data sources / hand-tuned text).
• Multi-indicator views dominate. FAUST cannot live on any single indicator when a view shows multiple indicators — you have to write it explicitly per view in the explorer YAML (or build views via c.group_views(...)).
• Single-shot migration with no plan to maintain. The duplication of full-YAML doesn't matter if no one will edit it again.

Step 3 — The Python step

Full-YAML variant

"""<one-line description of what this explorer surfaces>."""

from etl.helpers import PathFinder

paths = PathFinder(__file__)


def run() -> None:
    config = paths.load_collection_config()
    c = paths.create_collection(
        config=config,
        short_name="<short-with-hyphens>",  # explorer slug
        explorer=True,
    )
    c.save(tolerate_extra_indicators=True)

tolerate_extra_indicators=True is the common case: the upstream grapher dataset usually carries more indicators than the explorer references, and without this flag c.save() errors on the unused ones.

Table-driven variant

"""<one-line description>."""

from etl.helpers import PathFinder

paths = PathFinder(__file__)

# Map column → dimension tuple. "na" is the conventional empty slot for conditional dimensions
# (e.g. cost_metric is meaningful only when type=cost; affordability views set cost_metric="na").
COLUMN_DIMENSIONS: dict[str, dict[str, str]] = {
    "<col_a>": {"dim1": "value_a1", "dim2": "value_a2"},
    "<col_b>": {"dim1": "value_b1", "dim2": "value_b2"},
    # ...
}


def run() -> None:
    config = paths.load_collection_config()

    ds = paths.load_dataset("<grapher_dataset>")
    tb = ds.read("<table>", load_data=False)  # metadata only — faster, we don't need values

    for column, dims in COLUMN_DIMENSIONS.items():
        tb[column].m.dimensions = dims
        tb[column].m.original_short_name = "<unifying_indicator_name>"

    c = paths.create_collection(
        config=config,
        tb=tb,
        indicator_names=["<unifying_indicator_name>"],
        dimensions={
            "dim1": ["value_a1", "value_b1", ...],   # explicit choice order
            "dim2": ["value_a2", "value_b2", ...],
        },
        # common_view_config={...},                   # only if not in indicator metadata
        short_name="<short-with-hyphens>",
        explorer=True,
    )

    # Optional post-processing — see "Post-processing" below.
    # c.sort_choices({"dim1": lambda x: sorted(x)})
    # c.group_views([...])

    c.save(tolerate_extra_indicators=True)

Key APIs (see etl/collection/core/expand.py and etl/collection/core/create.py):

• tb[col].m.dimensions: dict[str, str] — required per column. Each entry says "this column represents the (dim1=value, dim2=value) cell." Columns without m.dimensions are ignored by the expander.
• tb[col].m.original_short_name: str — the unifying indicator name. With indicator_names=[that_name] and a single name, the expander treats all N columns as one logical indicator with N dimension combinations and drops the auto-added "indicator" pseudo-dimension.
• dimensions= accepts:
  • None → all dimensions found, arbitrary order.
  • list[str] → restricts and orders dimensions, all values shown.
  • dict[str, list[str] | "*"] → restricts and orders both dimensions and choices. Use "*" for "all values, arbitrary order."
• common_view_config= is applied uniformly to every auto-expanded view. Use it for fields that are truly shared and don't live at indicator level. Prefer indicator-level presentation.grapher_config for anything that should also flow to standalone charts.

Step 4 — The config YAML

Always block style. Mappings and lists in explorer config YAML must use block style — one key per line, list items on their own line under -. Never use flow style ({ key: value, ... } or [a, b, c]) even for tiny per-view dimensions: blocks. PR review on a 45-view file is unreadable when half the views collapse to a single flow line. The only exception is markdown links inside a quoted-scalar subtitle:/note: (those [text](url) brackets are content, not YAML structure).

config:
  # Explorer settings rows — keys map verbatim from the legacy TSV settings section.
  explorerTitle: ...
  explorerSubtitle: ...
  isPublished: true
  hasMapTab: false
  hideAlertBanner: true
  hideAnnotationFieldsInTitle: true
  entityType: country         # or "food", "region", etc.
  thumbnail: https://assets.ourworldindata.org/uploads/...
  wpBlockId: "12345"
  subNavId: explorers
  subNavCurrentId: <slug>
  selection:
    - <default selected entity>
    - <another>
  pickerColumnSlugs: []        # an empty list is OK; non-empty must be block-style
  yAxisMin: 0
  # ...

definitions:
  # Shared config applied to all views. Use this list (with optional `dimensions:`
  # filter per entry) — NOT YAML anchors and `<<:` merge keys. The framework merges
  # entries at expansion time; per-view `config:` blocks override anything here.
  common_views:
    - config:
        type: DiscreteBar
        hasMapTab: false
    # Dimension-filtered overrides apply only to matching views:
    # - dimensions:
    #     metric: share
    #   config:
    #     note: "Share values sum to 100%"

dimensions:
  # one entry per dropdown / radio / checkbox the user toggles
  - slug: <snake_case>          # e.g. "metric"
    name: <human label>          # e.g. "Metric"
    presentation:
      type: dropdown             # or radio / checkbox
    choices:
      - slug: <choice_snake>
        name: "<as shown in widget>"
      - slug: <another>
        name: "..."

views:
  # one entry per (dim1=x, dim2=y, …) tuple
  - dimensions:
      <dim_slug>: <choice_slug>
      # ...
    indicators:
      y:
        - catalogPath: <table>#<short>      # short form — see "catalogPath — short forms accepted" below
          display:                          # per-view, per-indicator overrides
            colorScaleNumericBins: 0;1;2
            colorScaleScheme: PuBu
    config:
      # Only per-view overrides here. Common stuff lives in definitions.common_views.
      # No `<<:` merge keys, no `&anchor`s.
      title: ...
      subtitle: ...
      type: <chart type>         # LineChart, DiscreteBar, "LineChart DiscreteBar", StackedArea, …
      hasMapTab: false
      minTime: 1990
      yAxisMin: 0

For table-driven explorers, views: should still be present but is typically views: [] — the explorer JSON schema requires the key, and create_collection(tb=tb, ...) populates the views at runtime.

catalogPath — short forms accepted

The Indicator.is_a_valid_path check (etl/collection/model/view.py:62) accepts three forms; pick the shortest one that still unambiguously resolves:

Short forms are expanded at c.save() time by Indicator.expand_path(tables_by_name). If the table name doesn't exist in any dependency it raises Table name '<x>' not found in dependency tables; if multiple dependencies expose the same table name, it raises and asks you to disambiguate with the medium form.

Default to table#indicator when authoring YAML. The full path is verbose, drifts when upstream versions bump, and is only needed for genuinely ambiguous cases.

Top-level config: settings — the most common keys

Dimension presentation types

• dropdown: shown as <select>. Use for >4 choices or when the choices have long labels.
• radio: shown as a row of pills. Use for ≤4 mutually-exclusive choices.
• checkbox: shown as a single toggle. Two choices only — usually "off" (slug like combined/absolute/no) and "on" (slug like the field name). Pair with presentation.choice_slug_true: <on_slug> so the framework knows which slug means "checked."

- slug: by_stage
  name: By stage of supply chain
  presentation:
    type: checkbox
    choice_slug_true: stages
  choices:
    - slug: combined
      name: ""
    - slug: stages
      name: By stage of supply chain

Conditional dimensions (the "na" pattern)

When a dimension is only meaningful for some rows (e.g. cost_metric matters only when type=cost, not when type=affordability), include the dimension everywhere with an "na" slot:

• Tag the columns/views that don't use it with <dim>: "na".
• Declare the na choice with name: "" so the widget renders as empty when applicable.
• Model: agriculture/latest/food_prices.{py,config.yml}.

- slug: cost_metric
  name: Cost metric
  presentation:
    type: radio
  choices:
    - slug: na
      name: ""
    - slug: dollars_per_day
      name: $ per day

Step 5 — Push FAUST upstream (recommended for single-indicator views)

For single-indicator views, the rendered chart inherits the indicator's stored grapher_config from MySQL at render time (both standalone-chart and explorer-view paths). Push:

For the chart-heading flow specifically, the priority is grapher_config.title > title_public > display.name > title (see docs/architecture/metadata/faqs.md). When migrating a chart-wrapping explorer, the chart's bespoke heading text belongs in grapher_config.title — title_public is the human-readable replacement for a dimensional indicator.title, not the chart's heading.

Cross-cutting baselines (e.g. hasMapTab: true for every indicator in a dataset) go under definitions.common.presentation.grapher_config in the garden .meta.yml. The catalog merge is recursive on presentation and grapher_config (lib/catalog/owid/catalog/core/yaml_metadata.py:_merge_variable_metadata), so each indicator inherits the common defaults plus its own overrides without manual <<: *anchor repetition.

DRY for repeated text fragments via dynamic-yaml interpolation:

definitions:
  prefix: "Long shared phrase about diet X."
  suffix: "Common closing sentence about methodology."

# In the indicator block:
subtitle: "{definitions.prefix} {definitions.suffix}"

This composes N unique full strings from a handful of building blocks. Verified via dynamic_yaml_to_dict (lib/catalog/owid/catalog/core/utils.py).

Step 6 — Post-processing the collection (table-driven only)

After paths.create_collection() returns the collection c, you can mutate it before c.save():

• c.sort_choices({dim_slug: lambda x: sorted(x)}) — control the order of dimension dropdowns. Useful when slugs sort poorly alphabetically.

• c.group_views(groups=[...]) — bundle multiple existing views into a new multi-indicator view. Each entry in groups:

  • dimension: the dimension whose choices are being collapsed.
  • choices: the choice slugs to combine (omit for all).
  • choice_new_slug: name for the new collapsed choice (e.g. combined, total, breakdown).
  • view_config: chart-level config for the new view (chartTypes, title, subtitle, selectedFacetStrategy, …). Title can be a template like "Population aged {age}" evaluated against params.
  • view_metadata: data-page metadata (description_key, etc.) for the new view.
  • replace=True to drop the originals; default keeps both.
  • overwrite_dimension_choice=True if choice_new_slug collides with an existing choice and you want grouped views to win.
  • Use cases: an explorer with sex={female, male} views — group_views adds sex=combined showing both timeseries on one chart. Same pattern works for age brackets, region groups, conflict types, or any dimension where users may want a single multi-line chart.

• c.edit_views([...]) — apply chart-level config to many views at once, optionally scoped by dimension. Each entry is {"dimensions": <filter>, "config": {...}, "metadata": {...}}; the framework merges entries by specificity (more dimensions in the filter = wins on conflicts). Use this in preference to set_global_config whenever you have per-slice overrides:

  c.edit_views([
      # No filter → applies to every view (defaults).
      {"config": {"type": "LineChart DiscreteBar", "hasMapTab": True}},
      # Scoped override — only this exact (gas, accounting, fuel, count) cell gets a Slope tab.
      {
          "dimensions": {"gas": "co2", "accounting": "territorial", "fuel": "all_fossil", "count": "per_capita"},
          "config": {"type": "LineChart SlopeChart DiscreteBar"},
      },
  ])
  

  Callable values inside config (e.g. "title": lambda v: ...) are evaluated against each matching view, so dimension-aware text templates still work. set_global_config is just a one-entry shortcut for edit_views; reach for edit_views once you have more than one slice to address.

• Manual loop over c.views — for things edit_views can't reach: per-indicator display blocks (numDecimalPlaces, colorScaleScheme, colorScaleNumericBins, display.color, …). These live on view.indicators.y[i].display, not on view.config. Match views via the view.matches(**kwargs) helper, which accepts a single value or a list (list = OR semantics):

  for view in c.views:
      if view.matches(gas=["methane", "all_ghg"], count="per_capita"):
          decimals = 1
      elif view.matches(gas="warming_impact", fuel=["land_use", "fossil_plus_land_use"]):
          decimals = 3
      else:
          continue
      for indicator in view.indicators.y or []:
          indicator.display = {**(indicator.display or {}), "numDecimalPlaces": decimals}
  

  Pattern: migration_flows.py's add_display_settings(c). Avoid this loop when the setting can live on the indicator's garden metadata instead.

• choice_renames={dim: {slug: display_name, ...}} (passed directly to create_collection) — map slug → display name when you need to derive the display label programmatically. Model: chart/minerals/latest/minerals.py.

• Sidecar <short>.dims.yaml — when the column → dimensions map exceeds ~50 entries, lift it out of the Python step into a sidecar YAML loaded at module-import time. Keeps <short>.py focused on logic and turns dim-tagging changes into a 1-line YAML edit. Model: etl/steps/viz/explorer/emissions/latest/co2.{py,dims.yaml}:

  from pathlib import Path
  import yaml
  COLUMN_DIMENSIONS = yaml.safe_load((Path(__file__).parent / "co2.dims.yaml").read_text())
  

Step 7 — DAG entry

In dag/<ns>.yml:

viz://explorer/<ns>/latest/<short>:
  - data://grapher/<ns1>/<v1>/<dataset1>
  - data://grapher/<ns2>/<v2>/<dataset2>
  # ... one line per unique upstream grapher dataset

Place near related explorer entries (or alongside the upstream grapher steps) for discoverability.

Step 8 — Verify

Hand off to the user:

1. .venv/bin/etlr viz://explorer/<ns>/latest/<short> --grapher — runs the step and upserts the explorer to the staging DB.
2. Open http://staging-site-<branch>/admin/explorers/preview/<slug> and spot-check:
   • default view (no dimensions toggled)
   • every dimension switch
   • map tab (if applicable)
   • country/entity picker
   • default selection
3. For migrations: diff the resulting TSV against the legacy owid-grapher/explorers/<slug>.explorer.tsv. Cosmetic differences (column ordering, whitespace) are acceptable; structural differences (missing views, swapped dimension orderings) are not. The Wizard's apps/wizard/app_pages/explorer_diff/ page does this comparison interactively for staging vs production.
4. make check.

Hand the user the exact etlr command — don't run it yourself.

Common pitfalls

• build_views does not propagate per-view display from indicator metadata — see TODO at etl/collection/core/expand.py:313. Each auto-expanded view gets indicators.y[0] with only catalogPath, no display. If you need different color scales per view, either (a) put them in the indicator's presentation.grapher_config.map.colorScale so they apply at chart render time, or (b) post-process c.views in Python.
• type: LineChart is not a valid grapher_config field when authoring via indicator metadata — use chartTypes: ["LineChart"] (the schema is an array). Per-view config.type in the explorer YAML still accepts strings.
• tb.read(..., load_data=False) is essential when you only need column metadata to set dimensions; loading data unnecessarily slows the step.
• YAML schema requires views: key in the explorer config even when empty. Pass views: [].
• Indicator must be re-published to MySQL with the new grapher_config before the explorer view can inherit it. Re-run etlr --grapher data://grapher/... after editing garden metadata; the explorer step alone won't refresh the indicator's stored config.
• YAML anchors / merge keys (&common_view, <<: *common_view) — don't use them. They can't filter by dimension and add per-view noise. Use definitions.common_views instead.
• Hyphens in short_name — file is food_footprints.py but short_name="food-footprints". Mismatch produces a published explorer whose URL doesn't match the legacy slug.
• tolerate_extra_indicators — usually want True for explorers since you're cherry-picking indicators from larger upstream datasets.
• Checkbox dimensions must have exactly 2 choices. The na pattern (3 choices: na, off, on) works for radio and dropdown but not checkbox — the framework rejects it with Dimension choices for 'checkbox' must have exactly two choices. If you genuinely need a third "doesn't apply" state, either use a radio with three choices, or drop the na slot and let the framework hide/disable the toggle when the current dimension state has no matching <true> view.
• YAML 1.1 booleanizes unquoted no/yes/on/off/true/false slugs. A view.dimensions: { relative_to_world: no } reads back as {"relative_to_world": False}, which won't match the string slug "no" declared in dimensions[*].choices. Use semantic slugs (total/relative, absolute/share) or quote the strings — but choosing different slug names is the more durable fix.
• edit_views doesn't reach indicator-level display. Fields like numDecimalPlaces, colorScaleScheme, colorScaleNumericBins, and per-indicator color live on view.indicators.y[i].display, not on view.config. edit_views only writes view-level config/metadata. For these, fall back to a c.views loop (see Step 6) or push them into the indicator's garden presentation.grapher_config so they apply at chart render time.

Reference examples

Full-YAML (each view hand-listed):

• etl/steps/viz/explorer/agriculture/latest/crop_yields.{py,config.yml} — large indicator-based explorer with many dimensions.
• etl/steps/viz/explorer/agriculture/latest/food_prices.{py,config.yml} — small grapher-chart-based migration (12 chart IDs unwrapped to 12 single-indicator views) with conditional dimensions ("na" pattern).
• etl/steps/viz/explorer/agriculture/latest/fertilizers.{py,config.yml} — checkbox dimension with choice_slug_true; multi-namespace dependencies.
• etl/steps/viz/explorer/food/latest/food_footprints.{py,config.yml} — hybrid (16 grapher-chart views + 29 CSV-backed views) showing dimension-filtered common_views for differing sourceDesc per view-type.
• etl/steps/viz/explorer/war/latest/countries_in_conflict_data.{py,config.yml} — uses na-named choices to model conditional dimensions.
• etl/steps/viz/explorer/emissions/latest/ipcc_scenarios.{py,config.yml} — moderate-size YAML-driven.

Table-driven (views auto-expanded from a dimensional table):

• etl/steps/viz/explorer/migration/latest/migration_flows.{py,config.yml} — passes tb=tb, indicator_names=[...], dimensions=[...] to create_collection; YAML carries only the static config and dimension presentation. Includes add_display_settings(c) post-processing.
• etl/steps/viz/explorer/emissions/latest/co2.{py,dims.yaml,config.yml} — sidecar .dims.yaml for the 54-entry column→dimensions map; uses c.edit_views([...]) with both an unscoped default and a 4-dim-filtered override; uses a c.views loop with view.matches(...) for per-indicator numDecimalPlaces overrides.
• etl/steps/viz/explorer/emissions/latest/air_pollution.{py,config.yml} — table-driven with c.group_views(...) to add facet views, c.drop_views(...) to prune cross-products.
• etl/steps/viz/chart/minerals/latest/minerals.py — same APIs in the multidim channel; useful read for choice_renames.

Follow-up

Once an explorer is on create_collection(explorer=True), it's a candidate for the Track-B port to MDIM (viz://chart/...) once feature parity is reached. See umbrella issue #6014.