| name | create-explorer |
| description | Author or modify an Our World in Data explorer (multi-dimensional dashboard with dropdown selectors, published from ETL via `export://explorers/<ns>/latest/<short>`). Trigger when the user wants to build a new explorer, add/remove views or dimensions on an existing one, change the explorer's chart text or selection defaults, or finish an explorer migration once the snapshot/garden/grapher chain is already in place. |
| metadata | {"internal":true} |
Creating an Explorer
Explorers are OWID's multi-dimensional dashboards (e.g. ourworldindata.org/explorers/food-prices). They're authored as YAML in this repo and published by ETL at export://explorers/<ns>/latest/<short>.
This skill is the explorer-flavored sibling of /create-multidim. They use the same paths.create_collection(...) engine and the same YAML schema for dimensions / views / definitions.common_views. The differences are:
| Multidim | Explorer |
|---|
| Channel | export://multidim/... | export://explorers/... |
| Step file location | etl/steps/export/multidim/<ns>/latest/ | etl/steps/export/explorers/<ns>/latest/ |
create_collection flag | (default) | explorer=True |
| Top-level config block | title:, default_selection:, default_dimensions: | config: block carrying legacy explorer settings (explorerTitle, explorerSubtitle, selection, subNavId, entityType, …) |
| Slug convention | underscores in file paths and short_name | underscores in file path, hyphens in URL slug and short_name argument |
| Verification | preview URL on staging | preview URL on staging + diff against owid-grapher/explorers/<slug>.explorer.tsv |
| Save call | c.save() | c.save(tolerate_extra_indicators=True) (upstream grapher datasets usually have more indicators than the explorer references) |
If you're modifying an existing explorer (adjusting chart text, swapping a catalogPath, adding a dimension choice, reordering views), most of the deeper sections below don't apply — find the existing <short>.config.yml, edit, run etlr, done. The full structure is documented below for new explorers and substantial reshapes.
When to use this skill
- After a
/migrate-explorer-csv, /migrate-explorer-grapher, or /migrate-explorer-indicator-legacy skill has produced (or already located) the upstream snapshot/meadow/garden/grapher chain, and now needs the export step.
- For a brand-new explorer where the data is already in ETL (skip directly to step 1).
- When porting an existing explorer's view layout (e.g. full-YAML → table-driven, or moving FAUST text from per-view YAML up into indicator metadata).
Step 1 — Files & directories
Every explorer is exactly two files plus a DAG entry:
etl/steps/export/explorers/<ns>/latest/
├── <short>.py # Python uses snake_case
└── <short>.config.yml
mkdir -p etl/steps/export/explorers/<ns>/latest
Hyphens vs underscores (recurring source of confusion):
- The Python file path uses underscores:
food_footprints.py, crop_yields.py.
- The explorer slug used in the URL and the
short_name= argument keeps hyphens: food-footprints, crop-yields.
paths.create_collection(short_name="<short-with-hyphens>") — pass the hyphenated slug.
Step 2 — Pick a construction style
The two ends of the spectrum, plus everything in between:
| Full-YAML | Programmatic / table-driven |
|---|
| Where views live | Hand-listed in <short>.config.yml under views: | Auto-expanded by paths.create_collection(tb=tb, ...) from columns whose m.dimensions is set |
| Where chart text (FAUST) lives | Per-view view.config.{title, subtitle, note} in YAML | presentation.{title_public, grapher_config} on each indicator's garden metadata; common defaults via definitions.common.presentation.grapher_config |
Where chart-level config lives (hasMapTab, tab, yAxis, chartTypes) | Per-view view.config | Indicator's presentation.grapher_config — single source of truth, same as for any standalone chart on that indicator |
| Map color scale | view.indicators.y[i].display.{colorScaleScheme, colorScaleNumericBins} (semicolon-string form, explorer-flavored override) | presentation.grapher_config.map.colorScale.{baseColorScheme, binningStrategy, customNumericValues} (canonical grapher form, inherited at chart render time) |
| Python step content | Trivial: paths.create_collection(config=config, short_name=..., explorer=True).save(...) | Loops columns to set m.dimensions, optionally post-processes (sort_choices, group_views, per-view display tweaks) |
It's a spectrum, not a switch. Mix freely: use table-driven for the bulk of views, hand-list a handful of bespoke ones; or stay full-YAML but still push title/subtitle for the single-indicator views into garden metadata to remove duplication.
Strong fit for table-driven:
- Single-indicator views dominate. Each view is a thin wrapper around one indicator → that indicator's metadata is the right home for chart text. Avoids duplication between explorer YAML and the equivalent standalone chart, and keeps both in sync forever.
- Many views (>20) following the cartesian product of a few dimensions. Hand-listing them is repetitive; auto-expansion plus YAML dimensions is significantly less code.
- Upstream is a dimensional table (one row per country/year × dim_a × dim_b × …) with one indicator.
create_collection(tb=tb, indicator_names=..., dimensions=...) matches this shape directly — model: migration/latest/migration_flows.py.
- Same indicators back standalone grapher charts. Pushing FAUST upstream means explorer view and standalone chart inherit the same text — no drift over time.
Stick with full-YAML when:
- Few views (<10), all bespoke (different chart types / data sources / hand-tuned text).
- Multi-indicator views dominate. FAUST cannot live on any single indicator when a view shows multiple indicators — you have to write it explicitly per view in the explorer YAML (or build views via
c.group_views(...)).
- Single-shot migration with no plan to maintain. The duplication of full-YAML doesn't matter if no one will edit it again.
Step 3 — The Python step
Full-YAML variant
"""<one-line description of what this explorer surfaces>."""
from etl.helpers import PathFinder
paths = PathFinder(__file__)
def run() -> None:
config = paths.load_collection_config()
c = paths.create_collection(
config=config,
short_name="<short-with-hyphens>",
explorer=True,
)
c.save(tolerate_extra_indicators=True)
tolerate_extra_indicators=True is the common case: the upstream grapher dataset usually carries more indicators than the explorer references, and without this flag c.save() errors on the unused ones.
Table-driven variant
"""<one-line description>."""
from etl.helpers import PathFinder
paths = PathFinder(__file__)
COLUMN_DIMENSIONS: dict[str, dict[str, str]] = {
"<col_a>": {"dim1": "value_a1", "dim2": "value_a2"},
"<col_b>": {"dim1": "value_b1", "dim2": "value_b2"},
}
def run() -> None:
config = paths.load_collection_config()
ds = paths.load_dataset("<grapher_dataset>")
tb = ds.read("<table>", load_data=False)
for column, dims in COLUMN_DIMENSIONS.items():
tb[column].m.dimensions = dims
tb[column].m.original_short_name = "<unifying_indicator_name>"
c = paths.create_collection(
config=config,
tb=tb,
indicator_names=["<unifying_indicator_name>"],
dimensions={
"dim1": ["value_a1", "value_b1", ...],
"dim2": ["value_a2", "value_b2", ...],
},
short_name="<short-with-hyphens>",
explorer=True,
)
c.save(tolerate_extra_indicators=True)
Key APIs (see etl/collection/core/expand.py and etl/collection/core/create.py):
tb[col].m.dimensions: dict[str, str] — required per column. Each entry says "this column represents the (dim1=value, dim2=value) cell." Columns without m.dimensions are ignored by the expander.
tb[col].m.original_short_name: str — the unifying indicator name. With indicator_names=[that_name] and a single name, the expander treats all N columns as one logical indicator with N dimension combinations and drops the auto-added "indicator" pseudo-dimension.
dimensions= accepts:
None → all dimensions found, arbitrary order.
list[str] → restricts and orders dimensions, all values shown.
dict[str, list[str] | "*"] → restricts and orders both dimensions and choices. Use "*" for "all values, arbitrary order."
common_view_config= is applied uniformly to every auto-expanded view. Use it for fields that are truly shared and don't live at indicator level. Prefer indicator-level presentation.grapher_config for anything that should also flow to standalone charts.
Step 4 — The config YAML
Always block style. Mappings and lists in explorer config YAML must use block style — one key per line, list items on their own line under -. Never use flow style ({ key: value, ... } or [a, b, c]) even for tiny per-view dimensions: blocks. PR review on a 45-view file is unreadable when half the views collapse to a single flow line. The only exception is markdown links inside a quoted-scalar subtitle:/note: (those [text](url) brackets are content, not YAML structure).
config:
explorerTitle: ...
explorerSubtitle: ...
isPublished: true
hasMapTab: false
hideAlertBanner: true
hideAnnotationFieldsInTitle: true
entityType: country
thumbnail: https://assets.ourworldindata.org/uploads/...
wpBlockId: "12345"
subNavId: explorers
subNavCurrentId: <slug>
selection:
- <default selected entity>
- <another>
pickerColumnSlugs: []
yAxisMin: 0
definitions:
common_views:
- config:
type: DiscreteBar
hasMapTab: false
dimensions:
- slug: <snake_case>
name: <human label>
presentation:
type: dropdown
choices:
- slug: <choice_snake>
name: "<as shown in widget>"
- slug: <another>
name: "..."
views:
- dimensions:
<dim_slug>: <choice_slug>
indicators:
y:
- catalogPath: <table>#<short>
display:
colorScaleNumericBins: 0;1;2
colorScaleScheme: PuBu
config:
title: ...
subtitle: ...
type: <chart type>
hasMapTab: false
minTime: 1990
yAxisMin: 0
For table-driven explorers, views: should still be present but is typically views: [] — the explorer JSON schema requires the key, and create_collection(tb=tb, ...) populates the views at runtime.
catalogPath — short forms accepted
The Indicator.is_a_valid_path check (etl/collection/model/view.py:62) accepts three forms; pick the shortest one that still unambiguously resolves:
| Form | Example | When to use |
|---|
table#indicator | global_carbon_budget#emissions_total | Default. Resolved against the explorer's DAG dependencies via tables_by_name — fine as long as no two dependencies expose a table with the same name. |
dataset/table#indicator | global_carbon_budget/global_carbon_budget#emissions_total | When two upstream datasets happen to expose tables with the same short_name. |
grapher/<ns>/<v>/<dataset>/<table>#<indicator> | grapher/gcp/2025-11-13/global_carbon_budget/global_carbon_budget#emissions_total | Only when you need to pin a specific dataset version separate from the one in the DAG — almost never the right form to write by hand. |
Short forms are expanded at c.save() time by Indicator.expand_path(tables_by_name). If the table name doesn't exist in any dependency it raises Table name '<x>' not found in dependency tables; if multiple dependencies expose the same table name, it raises and asks you to disambiguate with the medium form.
Default to table#indicator when authoring YAML. The full path is verbose, drifts when upstream versions bump, and is only needed for genuinely ambiguous cases.
Top-level config: settings — the most common keys
| Key | Type | Notes |
|---|
explorerTitle | string | Page title above the explorer. |
explorerSubtitle | string | One-liner under the title. |
isPublished | bool | true to publish; false keeps it draft. |
hasMapTab | bool | Whether any view shows the map tab by default. |
entityType | string | country (default), or food, region, species, etc. — controls picker labels. |
selection | list[str] | Default selected entities. |
pickerColumnSlugs | list[str] | Picker columns shown alongside the entity name. |
subNavId | string | Almost always explorers. |
subNavCurrentId | string | The slug — appears as the active nav item. |
wpBlockId | string | WordPress block ID for embedding (legacy). Stringify even when numeric. |
thumbnail | string | URL of preview image. |
hideAlertBanner | bool | Suppress the OWID-wide banner. |
hideAnnotationFieldsInTitle | bool | Drop time/entity from auto-titles. |
yAxisMin | number/string | Default Y-axis floor. |
yScaleToggle | bool | Allow user to toggle linear/log. |
originUrl | string | Path back to the topic page (e.g. /environmental-impacts-of-food). |
Dimension presentation types
dropdown: shown as <select>. Use for >4 choices or when the choices have long labels.
radio: shown as a row of pills. Use for ≤4 mutually-exclusive choices.
checkbox: shown as a single toggle. Two choices only — usually "off" (slug like combined/absolute/no) and "on" (slug like the field name). Pair with presentation.choice_slug_true: <on_slug> so the framework knows which slug means "checked."
- slug: by_stage
name: By stage of supply chain
presentation:
type: checkbox
choice_slug_true: stages
choices:
- slug: combined
name: ""
- slug: stages
name: By stage of supply chain
Conditional dimensions (the "na" pattern)
When a dimension is only meaningful for some rows (e.g. cost_metric matters only when type=cost, not when type=affordability), include the dimension everywhere with an "na" slot:
- Tag the columns/views that don't use it with
<dim>: "na".
- Declare the
na choice with name: "" so the widget renders as empty when applicable.
- Model:
agriculture/latest/food_prices.{py,config.yml}.
- slug: cost_metric
name: Cost metric
presentation:
type: radio
choices:
- slug: na
name: ""
- slug: dollars_per_day
name: $ per day
Step 5 — Push FAUST upstream (recommended for single-indicator views)
For single-indicator views, the rendered chart inherits the indicator's stored grapher_config from MySQL at render time (both standalone-chart and explorer-view paths). Push:
| Per-view config | → indicator garden metadata |
|---|
title | presentation.grapher_config.title |
subtitle | presentation.grapher_config.subtitle |
note | presentation.grapher_config.note |
map.colorScale | presentation.grapher_config.map.colorScale.{baseColorScheme, binningStrategy, customNumericValues} |
hasMapTab, tab, yAxis, chartTypes, hideRelativeToggle, selectedFacetStrategy, … | presentation.grapher_config.<field> |
For the chart-heading flow specifically, the priority is grapher_config.title > title_public > display.name > title (see docs/architecture/metadata/faqs.md). When migrating a chart-wrapping explorer, the chart's bespoke heading text belongs in grapher_config.title — title_public is the human-readable replacement for a dimensional indicator.title, not the chart's heading.
Cross-cutting baselines (e.g. hasMapTab: true for every indicator in a dataset) go under definitions.common.presentation.grapher_config in the garden .meta.yml. The catalog merge is recursive on presentation and grapher_config (lib/catalog/owid/catalog/core/yaml_metadata.py:_merge_variable_metadata), so each indicator inherits the common defaults plus its own overrides without manual <<: *anchor repetition.
DRY for repeated text fragments via dynamic-yaml interpolation:
definitions:
prefix: "Long shared phrase about diet X."
suffix: "Common closing sentence about methodology."
subtitle: "{definitions.prefix} {definitions.suffix}"
This composes N unique full strings from a handful of building blocks. Verified via dynamic_yaml_to_dict (lib/catalog/owid/catalog/core/utils.py).
Step 6 — Post-processing the collection (table-driven only)
After paths.create_collection() returns the collection c, you can mutate it before c.save():
-
c.sort_choices({dim_slug: lambda x: sorted(x)}) — control the order of dimension dropdowns. Useful when slugs sort poorly alphabetically.
-
c.group_views(groups=[...]) — bundle multiple existing views into a new multi-indicator view. Each entry in groups:
dimension: the dimension whose choices are being collapsed.
choices: the choice slugs to combine (omit for all).
choice_new_slug: name for the new collapsed choice (e.g. combined, total, breakdown).
view_config: chart-level config for the new view (chartTypes, title, subtitle, selectedFacetStrategy, …). Title can be a template like "Population aged {age}" evaluated against params.
view_metadata: data-page metadata (description_key, etc.) for the new view.
replace=True to drop the originals; default keeps both.
overwrite_dimension_choice=True if choice_new_slug collides with an existing choice and you want grouped views to win.
- Use cases: an explorer with
sex={female, male} views — group_views adds sex=combined showing both timeseries on one chart. Same pattern works for age brackets, region groups, conflict types, or any dimension where users may want a single multi-line chart.
-
c.edit_views([...]) — apply chart-level config to many views at once, optionally scoped by dimension. Each entry is {"dimensions": <filter>, "config": {...}, "metadata": {...}}; the framework merges entries by specificity (more dimensions in the filter = wins on conflicts). Use this in preference to set_global_config whenever you have per-slice overrides:
c.edit_views([
{"config": {"type": "LineChart DiscreteBar", "hasMapTab": True}},
{
"dimensions": {"gas": "co2", "accounting": "territorial", "fuel": "all_fossil", "count": "per_capita"},
"config": {"type": "LineChart SlopeChart DiscreteBar"},
},
])
Callable values inside config (e.g. "title": lambda v: ...) are evaluated against each matching view, so dimension-aware text templates still work. set_global_config is just a one-entry shortcut for edit_views; reach for edit_views once you have more than one slice to address.
-
Manual loop over c.views — for things edit_views can't reach: per-indicator display blocks (numDecimalPlaces, colorScaleScheme, colorScaleNumericBins, display.color, …). These live on view.indicators.y[i].display, not on view.config. Match views via the view.matches(**kwargs) helper, which accepts a single value or a list (list = OR semantics):
for view in c.views:
if view.matches(gas=["methane", "all_ghg"], count="per_capita"):
decimals = 1
elif view.matches(gas="warming_impact", fuel=["land_use", "fossil_plus_land_use"]):
decimals = 3
else:
continue
for indicator in view.indicators.y or []:
indicator.display = {**(indicator.display or {}), "numDecimalPlaces": decimals}
Pattern: migration_flows.py's add_display_settings(c). Avoid this loop when the setting can live on the indicator's garden metadata instead.
-
choice_renames={dim: {slug: display_name, ...}} (passed directly to create_collection) — map slug → display name when you need to derive the display label programmatically. Model: multidim/minerals/latest/minerals.py.
-
Sidecar <short>.dims.yaml — when the column → dimensions map exceeds ~50 entries, lift it out of the Python step into a sidecar YAML loaded at module-import time. Keeps <short>.py focused on logic and turns dim-tagging changes into a 1-line YAML edit. Model: etl/steps/export/explorers/emissions/latest/co2.{py,dims.yaml}:
from pathlib import Path
import yaml
COLUMN_DIMENSIONS = yaml.safe_load((Path(__file__).parent / "co2.dims.yaml").read_text())
Step 7 — DAG entry
In dag/<ns>.yml:
export://explorers/<ns>/latest/<short>:
- data://grapher/<ns1>/<v1>/<dataset1>
- data://grapher/<ns2>/<v2>/<dataset2>
Place near related explorer entries (or alongside the upstream grapher steps) for discoverability.
Step 8 — Verify
Hand off to the user:
.venv/bin/etlr export://explorers/<ns>/latest/<short> — runs the step and writes the TSV.
- Open
http://staging-site-<branch>/admin/explorers/preview/<slug> and spot-check:
- default view (no dimensions toggled)
- every dimension switch
- map tab (if applicable)
- country/entity picker
- default selection
- For migrations: diff the resulting TSV against the legacy
owid-grapher/explorers/<slug>.explorer.tsv. Cosmetic differences (column ordering, whitespace) are acceptable; structural differences (missing views, swapped dimension orderings) are not. The Wizard's apps/wizard/app_pages/explorer_diff/ page does this comparison interactively for staging vs production.
make check.
Hand the user the exact etlr command — don't run it yourself.
Common pitfalls
build_views does not propagate per-view display from indicator metadata — see TODO at etl/collection/core/expand.py:313. Each auto-expanded view gets indicators.y[0] with only catalogPath, no display. If you need different color scales per view, either (a) put them in the indicator's presentation.grapher_config.map.colorScale so they apply at chart render time, or (b) post-process c.views in Python.
type: LineChart is not a valid grapher_config field when authoring via indicator metadata — use chartTypes: ["LineChart"] (the schema is an array). Per-view config.type in the explorer YAML still accepts strings.
tb.read(..., load_data=False) is essential when you only need column metadata to set dimensions; loading data unnecessarily slows the step.
- YAML schema requires
views: key in the explorer config even when empty. Pass views: [].
- Indicator must be re-published to MySQL with the new
grapher_config before the explorer view can inherit it. Re-run etlr --grapher data://grapher/... after editing garden metadata; the explorer step alone won't refresh the indicator's stored config.
- YAML anchors / merge keys (
&common_view, <<: *common_view) — don't use them. They can't filter by dimension and add per-view noise. Use definitions.common_views instead.
- Hyphens in
short_name — file is food_footprints.py but short_name="food-footprints". Mismatch produces a published explorer whose URL doesn't match the legacy slug.
tolerate_extra_indicators — usually want True for explorers since you're cherry-picking indicators from larger upstream datasets.
- Checkbox dimensions must have exactly 2 choices. The
na pattern (3 choices: na, off, on) works for radio and dropdown but not checkbox — the framework rejects it with Dimension choices for 'checkbox' must have exactly two choices. If you genuinely need a third "doesn't apply" state, either use a radio with three choices, or drop the na slot and let the framework hide/disable the toggle when the current dimension state has no matching <true> view.
- YAML 1.1 booleanizes unquoted
no/yes/on/off/true/false slugs. A view.dimensions: { relative_to_world: no } reads back as {"relative_to_world": False}, which won't match the string slug "no" declared in dimensions[*].choices. Use semantic slugs (total/relative, absolute/share) or quote the strings — but choosing different slug names is the more durable fix.
edit_views doesn't reach indicator-level display. Fields like numDecimalPlaces, colorScaleScheme, colorScaleNumericBins, and per-indicator color live on view.indicators.y[i].display, not on view.config. edit_views only writes view-level config/metadata. For these, fall back to a c.views loop (see Step 6) or push them into the indicator's garden presentation.grapher_config so they apply at chart render time.
Reference examples
Full-YAML (each view hand-listed):
etl/steps/export/explorers/agriculture/latest/crop_yields.{py,config.yml} — large indicator-based explorer with many dimensions.
etl/steps/export/explorers/agriculture/latest/food_prices.{py,config.yml} — small grapher-chart-based migration (12 chart IDs unwrapped to 12 single-indicator views) with conditional dimensions ("na" pattern).
etl/steps/export/explorers/agriculture/latest/fertilizers.{py,config.yml} — checkbox dimension with choice_slug_true; multi-namespace dependencies.
etl/steps/export/explorers/food/latest/food_footprints.{py,config.yml} — hybrid (16 grapher-chart views + 29 CSV-backed views) showing dimension-filtered common_views for differing sourceDesc per view-type.
etl/steps/export/explorers/war/latest/countries_in_conflict_data.{py,config.yml} — uses na-named choices to model conditional dimensions.
etl/steps/export/explorers/emissions/latest/ipcc_scenarios.{py,config.yml} — moderate-size YAML-driven.
Table-driven (views auto-expanded from a dimensional table):
etl/steps/export/explorers/migration/latest/migration_flows.{py,config.yml} — passes tb=tb, indicator_names=[...], dimensions=[...] to create_collection; YAML carries only the static config and dimension presentation. Includes add_display_settings(c) post-processing.
etl/steps/export/explorers/emissions/latest/co2.{py,dims.yaml,config.yml} — sidecar .dims.yaml for the 54-entry column→dimensions map; uses c.edit_views([...]) with both an unscoped default and a 4-dim-filtered override; uses a c.views loop with view.matches(...) for per-indicator numDecimalPlaces overrides.
etl/steps/export/explorers/emissions/latest/air_pollution.{py,config.yml} — table-driven with c.group_views(...) to add facet views, c.drop_views(...) to prune cross-products.
etl/steps/export/multidim/minerals/latest/minerals.py — same APIs in the multidim channel; useful read for choice_renames.
Follow-up
Once an explorer is on create_collection(explorer=True), it's a candidate for the Track-B port to MDIM (export://multidim/...) once feature parity is reached. See umbrella issue #6014.