| name | migrate-explorer-indicator-legacy |
| description | Modernize an explorer step that already exists in this repo but doesn't use the YAML-driven `paths.create_collection(explorer=True)` API yet. Covers two legacy shapes: (A) `data://explorers/<ns>/<v>/<short>` steps that write wide CSV tables for the legacy explorer infra (e.g. poverty_inequality, lis, wid, wb, emdat, monkeypox); (B) `export://explorers/<ns>/latest/<short>` steps that already exist but build the TSV programmatically via `paths.create_explorer_legacy(df_graphers, df_columns)` (minerals, air_pollution, migration/2024-08-05, minerals_supply_and_demand_prospects). This skill extracts the dimensions/views/config from the legacy shape and hands them to `/create-explorer` to write the modern step. Trigger when the user says "migrate indicator-legacy explorer <short>", "convert create_explorer_legacy to create_collection", or "move data://explorers/... to export://explorers/...". |
| metadata | {"internal":true} |
Modernize a Legacy ETL Explorer Step
Take an explorer step that already lives in this repo but uses the old DataFrame-driven plumbing, extract the dimensions/views/config, and hand off to /create-explorer to write the modern YAML-driven step. Then remove the legacy code.
Scope of this skill: identifying which legacy shape you have, reading the legacy code, mapping its DataFrames into the dimension/view structure that /create-explorer consumes, and removing the old step. The export-step authoring (Python skeleton, YAML schema, full-YAML vs table-driven, FAUST upstream, post-processing, DAG, verification) lives in /create-explorer. Don't duplicate that content here.
Inputs
Required:
- The legacy step path. Either:
- Pattern A —
etl/steps/data/explorers/<ns>/<v>/<short>.py (writes wide CSV tables under data://explorers/...); or
- Pattern B —
etl/steps/export/explorers/<ns>/<v>/<short>.py that calls paths.create_explorer_legacy(...).
- Target
<ns> and <short> for the new step.
Not covered: explorers that don't have any ETL step yet (e.g. plastic-pollution, conflict-data, conflict-data-source, countries-in-conflict-data referenced in umbrella issue #6014). For those use /migrate-explorer-grapher (their TSVs reference indicator IDs and need fresh authoring as YAML, not modernization of an existing step).
Step 0 — Identify which pattern you have
ls etl/steps/data/explorers/
grep -l create_explorer_legacy etl/steps/export/explorers/ -r
| Pattern | Examples in repo today | What it produces |
|---|
A. data://explorers/<ns>/<v>/<short> | poverty_inequality, lis/luxembourg_income_study, wid/world_inequality_database, wb/world_bank_pip, emdat/natural_disasters, who/monkeypox, dummy | A wide CSV at data/explorers/<ns>/<v>/<short>/... that the live explorer TSV (owid-grapher/explorers/<short>.explorer.tsv) reads via tableSlug. |
B. paths.create_explorer_legacy(df_graphers, df_columns, ...) | minerals/latest/minerals, minerals/latest/minerals_supply_and_demand_prospects, emissions/latest/air_pollution, migration/2024-08-05/migration | A complete explorer TSV pushed straight to MySQL — no live TSV file in owid-grapher/, the ETL step is the source of truth. |
The two patterns share the destination (export://explorers/<ns>/latest/<short> with YAML config) but differ in starting state.
Pattern B workflow (most common today)
The legacy step builds two pandas DataFrames programmatically:
df_graphers — one row per view, with dimension columns (<Name> Dropdown / <Name> Radio / <Name> Checkbox), yVariableIds (or catalog-path strings), and per-view chart settings (hasMapTab, minTime, yAxisMin, defaultView, …).df_columns — per-(view, indicator) display overrides (color scales, tolerances, units, …). Sometimes omitted.
You're going to delete the DataFrame plumbing and replace it with a YAML config plus, optionally, a table-driven create_collection(tb=tb, ...) call when the dimensions map cleanly onto an upstream multidim grapher dataset.
1. Read the legacy step end-to-end
Identify:
- Static settings that go to
config: (title, subtitle, selection, isPublished, hasMapTab, …).
- The list of dimensions and their choices (the columns in
df_graphers that end with Dropdown/Radio/Checkbox, and the unique values per column).
- The mapping from (dimension choices) → (catalog path, per-view config). This is the bulk of the legacy code.
- Any post-processing that's hard to express in YAML (e.g. sorting choices, setting per-metric display defaults, conditional
hasMapTab=False). Worth noting so it can be ported as Python in the new step.
2. Map the legacy DataFrames to the explorer YAML structure
| Legacy code | Where it goes |
|---|
config = {...} dict in legacy run() | top-level config: block in YAML |
df_graphers["<Name> Dropdown"] unique values | one entry under dimensions: (slug = snake-case of <Name>, presentation type = dropdown/radio/checkbox) |
Each row of df_graphers | one entry under views: with dimensions: map and indicators.y[].catalogPath |
Per-row hasMapTab, minTime, yAxisMin, defaultView, type, … | view.config (or factored into definitions.common_views if shared across many views) |
Per-row title/subtitle/note for single-indicator views | Prefer presentation.grapher_config in the indicator's garden metadata; only put in view.config if the view has multiple indicators or the text genuinely differs from the indicator's metadata |
df_columns rows | view.indicators.y[].display (per-view, per-indicator) |
Loops setting display on every "Production" view, etc. | Either fold into definitions.common_views (with a dimension filter) or keep as Python post-processing on c.views |
A legacy step that already builds dimensions from a multidim grapher dataset (rather than hand-stitching variable IDs) is usually a clean fit for the table-driven variant in /create-explorer. If many of the legacy views are multi-indicator already, you'll either keep them in YAML or rebuild them via c.group_views(...) after auto-expansion (see /create-explorer Step 6).
3. Hand off to /create-explorer
Invoke /create-explorer with:
- the top-level
config: settings extracted in step 1
- the dimensions list (slugs, names, presentation types, choices)
- the views list (per-view dimension tuples + catalogPaths + per-view config overrides)
- any post-processing requirements you flagged in step 1 (
sort_choices, custom display loops)
- the upstream
data://grapher/... deps for the DAG entry — same as the legacy step's deps in dag/<ns>.yml
4. DAG
In dag/<ns>.yml, the /create-explorer skill writes the new export://explorers/<ns>/latest/<short>: block. You then need to:
- If a previous
export://explorers/<ns>/<v>/<short>: block existed (Pattern B with non-latest version), delete it from dag/<ns>.yml.
- For Pattern A: also remove the legacy
data://explorers/<ns>/<v>/<short>: block from dag/<ns>.yml. Find any consumers (grep -rn 'data://explorers/<ns>/<v>/<short>' dag/) and update them.
The archive dag (dag/archive/*.yml) is not edited by hand — etl archive-dag reconstructs it from git history and records the removed steps with their last-active commit (for recovery via git checkout).
5. Remove the legacy step
git rm etl/steps/export/explorers/<ns>/<v>/<short>.py
git rm etl/steps/data/explorers/<ns>/<v>/<short>.py
Pattern A workflow (data://explorers/<ns>/<v>/<short> writing wide CSVs)
This pattern is older. The legacy step produces a wide CSV with one column per (indicator, dimension) combination, and the live explorer TSV in owid-grapher/explorers/<short>.explorer.tsv reads from it via tableSlug blocks.
1. Read the legacy step
etl/steps/data/explorers/<ns>/<v>/<short>.py typically pivots a long table into wide form:
def run() -> None:
ds = paths.load_dataset("<garden_dataset>")
tb = ds.read("<table>")
tb_wide = tb.pivot(...)
ds_explorer = paths.create_dataset(tables=[tb_wide])
ds_explorer.save()
2. Read the live explorer TSV
owid-grapher/explorers/<short>.explorer.tsv defines dimensions, views, and per-indicator display configs. Use the same translation strategy as /migrate-explorer-grapher step 4:
- Settings rows → top-level
config: block.
graphers table → views: (one per row).columns table → per-view indicators[*].display.
3. Map column slugs in the wide CSV to grapher catalog paths
Each tableSlug/column reference in the live TSV corresponds to one wide-CSV column (metric_dimension1_dimension2). Find the underlying garden/grapher dataset that produces those indicators (the legacy step's input paths.load_dataset(...)), and rewrite each TSV column reference as a catalogPath pointing at that grapher dataset.
If indicators don't exist as separate grapher columns (e.g. the legacy CSV had a custom pivoted shape), you may need to either expose them in the upstream grapher step, or keep a thin "explorer table" grapher dataset.
4. Hand off to /create-explorer, then archive
Same as Pattern B steps 3–5.
Verify (both patterns)
Hand off to the user the verification commands per /create-explorer Step 8. For Pattern A specifically, also confirm the legacy data://explorers/... step is no longer being built (DAG-wise) and the live TSV in owid-grapher/ is no longer being read by the rendered explorer.
Reference: existing migrations
- Pattern B → table-driven:
etl/steps/export/explorers/migration/latest/migration_flows.py (the new shape that replaced migration/2024-08-05/migration.py).
- Pattern B → full-YAML:
etl/steps/export/explorers/agriculture/latest/crop_yields.{py,config.yml} shows the YAML-only style.
- Reference for legacy-to-modern diff: compare
migration/2024-08-05/migration.py (still on create_explorer_legacy) with migration/latest/migration_flows.py (modern).
Follow-up
Once on create_collection(explorer=True), the explorer is a candidate for the Track-B port to MDIM (export://multidim/...) once feature parity is reached. See umbrella issue #6014.
