End-to-end dataset update workflow with PR creation, snapshot, meadow, garden, and grapher steps. Use when user wants to update a dataset, refresh data, run ETL update, or mentions updating dataset versions.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
End-to-end dataset update workflow with PR creation, snapshot, meadow, garden, and grapher steps. Use when user wants to update a dataset, refresh data, run ETL update, or mentions updating dataset versions.
metadata
{"internal":true}
Update Dataset (PR → snapshot → steps → grapher)
Use this skill to run a complete dataset update with Claude Code subagents, keep a live progress checklist, and pause for user approval only when something needs attention.
Paired skill — keep in sync./review-data-pr is the reviewer-side counterpart of this skill: it verifies the outcomes of the author-side steps defined here. Whenever you add, remove, or change a workflow step in this file, check whether review-data-pr/SKILL.md needs a matching reviewer-side check (and add it in the same commit if so). The reverse also holds — see the mirror note there. The creation-side skills /create-dataset and /create-snapshot belong to the same family: they point into this file's canonical sections (§5b-bis sanity bounds, §5c harmonization audit, §6b metadata quality, §6c metadata checklist + link verification, §6d scheduled issues, and the snapshot-related Guardrails), so when one of those sections changes, check whether the create skills need a matching edit in the same commit too.
The two announcements are owned elsewhere./data-updates-comms owns the internal Slack form (step 9) and /data-update-announcement owns the public /latest post (step 9b). Both steps here are delegation stubs on purpose — put wording, format, and style guidance in those skills, never back into this file.
Inputs
<namespace>/<old_version>/<name>
Get <new_version> as today's date by running date -u +"%Y-%m-%d"
A bare <short_name> (no namespace/version) is also valid — it's what owid-issues reminder bodies use. Resolve it to <namespace>/<old_version>/<short_name> via the DAG: rg "/<short_name>:?$" dag/ -g "*.yml" | grep -v "^dag/archive" — the :? matters because active entries are YAML keys ending in : (a $-anchored pattern without it only matches dependency lines), and archived entries must never be resolution targets. Take the latest active version; ask the user if the short name is ambiguous across namespaces. Several space-separated short names (/update-dataset <short_name1> <short_name2>) mean a grouped update of related datasets: run the full workflow for each, on one shared branch/PR.
Read the dataset's scheduled-issue body before running etl update (step 6d locates the update-*.yml workflow — peek at it now, not only at 6d): it often carries grouped-update instructions naming companion datasets that must bump together, e.g. a deflator ("US PPI New Office Construction also needs to be updated alongside to adjust for inflation"). Also check the target's garden DAG entry for same-vintage dependency datasets (a data://garden/... dep sharing the old version date is usually a companion). Seed etl update with every companion snapshot so the chains move in one run — updating only the main dataset leaves the derived series (e.g. inflation-adjusted) with a silent NaN tail for the new months.
Optional trailing args:
branch: The working branch name (defaults to current branch)
Assumptions:
All artifacts are written to workbench/<short_name>/.
Persist progress to workbench/<short_name>/progress.md and update it after each step.
Persist reusable update facts to workbench/<short_name>/update-context.yml as they are discovered. This is the canonical context artifact for the PR description, review handoff, and data-updates-comms.
Progress checklist (maintain, tick live, and persist to progress.md)
Parse inputs and resolve: channel, namespace, version, short_name, old_version, branch
Run ETL update workflow via etl-update subagent (help → dry run → approval → real run)
Add yourself to dataset.owners in the new garden .meta.yml (don't reorder; preserve existing names and markers)
Catalog # NOTE: / # TODO: comments carried over from the old step files into notes_to_check.md
Detect any sanity_checks functions and their log-control flags; append to notes_to_check.md
Create or reuse draft PR and work branch
Update snapshot and compare to previous version; capture summary
Meadow step: run + fix + diff + summarize
Garden step: run + fix + diff + summarize
Surface new indicators: check meadow + garden diffs (and snapshot columns when meadow hardcodes a column subset) for new-version columns absent from the old; warn user + PR body, flag new-but-dropped columns, route rename pairs to the step-7 mapping
Review sanity_checks output (enable log flag, re-run, scan log, revert flag) — if none found and the garden step does non-trivial logic, recommend adding them; if present but missing value bounds (positive / [0,1] / [0,100] per indicator type), suggest those too (see 5b-bis)
Country harmonization audit: validate .countries.json against canonical regions (flag provider regions not yet in the regions dataset → /add-provider-regions), audit .excluded_countries.json, scan garden log for missing/unused/unknown warnings
Region-provider drift: if this dataset's aggregates are in (), check whether the new version changed the provider's region set or country membership; if so, update and re-propagate via
Persistence:
After ticking each item, update workbench/<short_name>/progress.md with the current checklist state and a timestamp.
Append every generalizable lesson to workbench/<short_name>/lessons.md the moment you hit it, not at the end. One entry per lesson: what you expected, what actually happened, and the general rule — plus which skill file should own it. Reconstructing these from memory after a long session loses the specifics that make a lesson usable (the exact column name, the threshold that separated signal from noise, the check that would have caught it earlier), and the ones worth keeping are exactly the ones that cost time mid-run. Fold them into the skills when the user asks, following feedback_skill_lessons_writeback conventions: phrase each as a general pattern with the incident as a trailing one-liner, and re-read the target skill first — a long-running branch can be several skill revisions behind master, so the section you remember may already have been rewritten by someone else's session.
Checkpoints — when to pause
Default: keep going. Run through the full workflow without stopping unless one of the conditions below is met.
Stop and ask the user when:
A step fails and the fix is ambiguous (multiple reasonable approaches, or you're unsure of the correct one)
Data structure changed significantly (columns removed/renamed, large row count drops, schema changes that may affect charts)
Country harmonization has new unmatched countries that need manual decisions
The snapshot requires a manual download or credentials you don't have
Indicator upgrade had imperfect matches (< 100% similarity) that need human review
Anything that could silently break charts or lose data
Don't stop for:
Routine assertion count updates (just update them and note in the summary)
Clean step runs with only row increases
Expected warnings (SettingWithCopyWarning, known unmapped territories)
When you do stop, present a concise summary of the issue and what options exist.
When the update isn't a drop-in version bump
Some updates carry structural changes that make the standard rename-only flow the wrong tool. Recognise them up front and adjust the workflow.
Triggers — any of these means you're in restructure territory, not a version bump:
short_name changes (producer rebranded the dataset).
File format/schema changes (wide → long, different file extension with a different column set, new dimensions).
Policy/indicator set changes substantially (splits, dropped composites, newly added areas).
Score semantics change (e.g. binary → continuous with subnational coverage).
Workflow adjustments:
Skip etl update. The rename-only flow copies the old step files into a new folder — useless when the schema is different. Author the new step chain by hand, using the old version as inspiration but not as a starting copy.
Add the new chain to the DAG before removing the old. Leave both chains active while you build and validate v2; remove the v1 entries only once v2 is on staging and the chart remap is queued or done.
Decide on naming convention upfront. Ask the user whether to preserve v1 short_names where they map cleanly, or to adopt the source's fresh naming scheme. Fresh naming is cleaner but means the auto-Indicator-Upgrader can't help.
Hand-curate the v1 → v2 indicator mapping. When short_names change entirely, the auto-upgrader has nothing to match on, but the Indicator Upgrader also matches on title — so if v2 titles are descriptive (full sentences rather than the bare short_name), you can hand the user a table of v1 title → v2 title pairs and they can drive the chart remap from there. Generate this table from the v1 meta.yml + the v2 grapher catalog.
Before remapping a chart onto a successor indicator, diff the chart's config against the new indicator's shape. Three things silently break on a v1 → v2 remap and none of them fails a build: (a) the chart's pinned selectedEntityNames may not exist in the successor (v1 steps often computed regional aggregates the producer doesn't publish — check the retired step's aggregation method via git show <archive-marker-sha>:<path> and rebuild the aggregates in garden the same way, e.g. population-weighted means); (b) pinned yAxis min/max sized to the old index's range can clip the new one; (c) the subtitle/FAUST may describe the old construction ("combines 18 policies") and become factually wrong. When the producer publishes several related successor indices, their codebook usually says which is the primary measure and which is a transparency baseline — cite that framing when picking the chart's indicator. (LGBTI: the v1 rights-index chart pinned World + six continents, an axis max of 13 against a successor peaking at 18.7, and a v1-specific subtitle — all three needed changes.)
Defer the Slack and /latest announcements until charts have been remapped. Both posts depend on charts.published_count and charts.selected_views from the v2 chain. Drafting them before the remap gives the wrong count (zero) and no representative views. Tell the user to ping you when the chart remap is done, then run steps 8 / 9 / 9b.
For the long-format with dimensions sub-case specifically (e.g. one row per (country, year, <dim1>, <dim2>)), use the modern OWID pattern:
Grapher: pass long tables through unchanged; the framework auto-expands them into per-cell variables.
Metadata: variables are keyed by the long-column name, with <% if <dim> == "X" and <dim2> == "Y" %>...<% endif %> Jinja blocks inside title, description_short, display.name. Grep this repo for tb.format(["country", "year" with more than two index entries to find current reference examples.
Jinja coverage: after building the grapher dataset, verify every active (dim1, dim2) cell renders a non-empty value — read every column from the built grapher dataset and assert metadata.title is non-empty. A dimension combination with no matching <% if %> branch ships an untitled indicator.
Workflow orchestration
Initial setup
Check if workbench/<short_name>/progress.md exists to determine if continuing existing update
If starting fresh: delete workbench/<short_name> directory if it exists
Create fresh workbench/<short_name> directory for artifacts
Run ETL update command (etl-update subagent)
Inputs: <namespace>/<old_version>/<short_name> plus any required flags
Pick the URI that matches what's actually changing:
If the source data is changing — new source files, modified extractor, anything that affects the snapshot output — run from the snapshot URI with --include-usages. This bumps the whole chain (snapshot → meadow → garden → grapher) to the new version together:
Foundational / widely-used datasets (e.g. wb/*/income_groups, regions, population): add --direct-only. Plain --include-usages follows usages transitively and would try to version-bump every downstream consumer (income_groups has ~85 across 15 dag files). --direct-only restricts the bump to steps sharing the dataset's own namespace/version/short_name, i.e. just its chain. Caveat: --direct-onlyexcludes sibling steps with a different short_name that belong to the same chain (e.g. income_groups_aggregations, which the grapher step also depends on) — pass those as extra seed steps so the grapher doesn't end up mixing a new-version garden with an old-version sibling. Dry-run and confirm the proposed set is exactly the chain before executing:
If only garden logic / metadata is changing and the source data is unchanged, run from the garden URI. This bumps garden and grapher only; snapshot and meadow stay on the old version.
Either way, run etl updateonce. Don't call it separately per channel — that leaves stale version references in the DAG (e.g., new garden pointing to old meadow).
Perform help check, dry run, approval, then real execution; capture summary for later PR notes
1a-bis) Add yourself to dataset.owners in the new garden .meta.yml
You've just become a contributor to this dataset, so add your canonical OWID name to its owners: list. Don't reorder — keep the existing primary first; append yourself at the end. Skip if you're already there.
Your canonical name must match an entry in the schema enum (schemas/dataset-schema.json). Resolve it from git config user.name via etl.owners.resolve_owner; if that returns None, add a mapping in etl/owners.py and a row in the schema enum before continuing.
Edit the YAML in place, preserving comments and the existing # review / # backport / # fasttrack markers on other entries.
1b) Check for outdated practices (check-outdated-practices skill)
After etl update creates new step files, run the /check-outdated-practices skill on every new step file — including helper modules that etl update doesn't generate but you copied by hand (e.g. *_omms.py), since those carry legacy patterns too
The skill reads the extension as the source of truth for the full pattern set (the geo.add_* aggregation/population helpers are flagged, not just geo.harmonize_countries) — don't rely on a remembered subset
Fix any findings before proceeding — this avoids propagating legacy patterns into new versions
geo.harmonize_countries → paths.regions.harmonize_names is mechanical and safe. geo.add_regions_to_table → paths.regions.add_aggregates changes the aggregation core — prove equivalence with a controlled A/B test, not a diff against the old feather. Build the new garden both ways against the same current catalog (swap the call, rebuild, save output; revert, rebuild, save output) and diff the two. Do NOT conclude "the helper shifts aggregates across all years" from a new-vs-old-feather diff — that conflates the helper with upstream-dataset drift (see step 5). In practice the two helpers are equivalent bar tiny historical edge cases (e.g. one region-year's population residual); if so, modernize. add_aggregates also auto-resolves income groups from the DAG, so it's the right tool when you later need WB income-group aggregates.
1c) Catalog # NOTE: / # TODO: comments in the copied step files (don't resolve yet)
Run rg -n "#\s*(NOTE|TODO|FIXME|HACK|XXX):" snapshots/<namespace>/<new_version>/ etl/steps/data/{meadow,garden,grapher}/<namespace>/<new_version>/.
Filter out generic boilerplate (e.g. # NOTE: To learn more about the fields, hover over their names. at the top of .meta.yml).
Save the remaining actionable items to workbench/<short_name>/notes_to_check.md — one entry per annotation, recording file path, line number, which step it lives in (meadow/garden/grapher), and what the workaround does.
Don't act on them yet. Resolution requires fresh data and happens after each step's run — see step 6a.
1d) Detect sanity-check logic in the copied step files
Sanity checks live in two different forms — detect both:
Function form — def sanity_check… / sanity_check…( call sites. Often gated by a module-level boolean flag (DEBUG, SHOW_SANITY_CHECK_LOGS, LONG_FORMAT) that defaults to False to keep normal runs quiet. Examples: etl/steps/data/garden/wb/.../world_bank_pip.py (SHOW_SANITY_CHECK_LOGS), etl/steps/data/garden/wid/.../world_inequality_database.py (DEBUG + LONG_FORMAT), etl/steps/data/garden/lis/.../luxembourg_income_study.py (no flag; prints unconditionally via tabulate).
Inline comment form — # Sanity check / # Sanity checks / # sanity check marking an inline assertion block that isn't wrapped in a dedicated function. Very common: etl/steps/data/garden/emdat/.../natural_disasters.py, etl/steps/data/garden/emissions/.../national_contributions.py, etl/steps/data/garden/irena/.../renewable_capacity_statistics.py. These usually have no log flag — the block simply runs on every step execution and either passes or raises.
Append a "Sanity checks" section to workbench/<short_name>/notes_to_check.md listing each hit — for each, record: file path + line number, which form (function vs. inline comment), the name of any log-control flag (function form only), and a one-line description of what's being asserted (read the surrounding 5–10 lines).
Don't act yet — the review happens in step 5b once the garden step has been run on the new data.
Create PR and integrate update via subagent (etl-pr)
Inputs: <namespace>/<old_version>/<short_name>
Create or reuse draft PR, set up work branch, and incorporate the ETL update outputs
Snapshot run & compare (snapshot-runner subagent)
Inputs: <namespace>/<new_version>/<short_name> and <old_version>
Hand-maintained snapshots + editorial data edits. Some snapshots have no url_download — the .py prompts for --path-to-file and the docstring says the data was "provided by email" / curated by hand. When the update is a small editorial correction (the user gives you the facts directly, e.g. "country X did Y in year Z"), you can produce the new snapshot yourself: copy the previous version's data file (data/snapshots/<ns>/<old_version>/<file>), change only the specific cells, and assert in a quick script exactly which rows/cells changed (and that all others are byte-identical) before running etls ... --path-to-file <edited>. Then update the .py docstring to document the edit and bump the .dvcdate_published / citation_full year (ask the user whether it's a new producer release or an OWID-applied edit — see step 6c). Verify the user's stated facts against the existing data first — some may already be encoded from a prior release (in this update, one of the two reported events was already in the live snapshot; only the other was a genuine change). Tell the user what's already present rather than blindly re-adding it.
Watch for meadow input checks keyed on absolute row/column positions. Producers quietly restructure their files (e.g. this session, the WB dropped the legend rows above the first country, shifting the row count 239→234 and moving "Afghanistan" from row 10 to row 5). Data extraction that keys off content (drop rows without an id, then melt) survives, but hardcoded tb.loc[N] / exact-row-count asserts break — update them to the new positions/counts and drop a # NOTE so the next maintainer re-checks. The break is the check doing its job; don't loosen it into uselessness.
Review the log: scan for AssertionError, error, warning, dropped, outliers flagged by country/year, unexpected totals. Surface actionable findings in the PR description under a "Sanity-check findings" collapsed section.
Revert the flag to its original value (usually False) before committing. Verify with git diff that the garden file has no unintended changes.
Function form with no flag, or inline # Sanity check(s) comment blocks:
Read each catalogued block (pull 5–15 lines of context around the hit) to understand what invariant is being tested.
Important: a sanity check can enforce its finding either by raising (assert, raise) or by logging (paths.log.warning, .critical, even .fatal). Logging variants do NOT fail the step — so "step 5 passed" is not proof that every invariant held. If the block uses logging, re-run the step and scan stdout/stderr for the relevant keywords; don't trust the exit code alone.
For non-trivial invariants (monotonicity, totals, bounds), also spot-check qualitatively against the fresh garden output via a short .venv/bin/python snippet.
Record any anomalies under "Sanity-check findings" in the PR description. No log artifact to keep here since the step's own output is the evidence.
In either form: if sanity_checks raise AssertionError on the new data, stop and decide with the user whether the assertion needs a threshold bump, whether upstream data genuinely broke, or whether the invariant is obsolete. If the check only logs, treat a new/expanding set of warnings the same way — they're the signal the sanity check was written to produce.
Watch for silent-delete patterns. Some sanity_checks functions also mutate the table — e.g. world_bank_pip's sanity_checks drops rows that fail invariants and reports the count via the log-control flag. With the flag off the deletions still happen; the reviewer just never learns which rows disappeared. When reading a sanity_checks function, scan for drop, filter, tb = tb[...] — anything that removes rows — and list every deletion in the PR body, not just the warning counts. If the deletion seems newly applicable to upstream fixes (e.g. the row should no longer be anomalous in the new release), that's a candidate for removing the workaround entirely.
5b-bis) Recommend sanity checks when the garden step lacks them (or lacks value bounds)
Runs after the garden step is built. Two triggers:
Step 1d catalogued no sanity checks and the garden step does more than a straight load-and-format (harmonization, column drops/renames, aggregations, derivations) — recommend adding sanity_check_inputs / sanity_check_outputs. A Codex review will flag this anyway; better to handle it during the update.
The step has checks but you can spot missing value-bound coverage of the kinds below — suggest the additions.
Ground every threshold in the built data before writing it. Run min/max/nunique/value_counts per indicator on the built garden table first — never write plausible-sounding bounds. The classic trap: "% of GDP" indicators look like percentages but legitimately exceed 100 (IMF PFMH: gross debt up to 495%, expenditure 595% in crisis years; UK's post-Napoleonic 260% debt is on a published chart). A blind 0–100 assert fails the build on day one.
Suggested checks by indicator type (propose the applicable ones, not all):
Indicator type
Check
Share / proportion
values in [0, 1]
Percentage of a whole
values in [0, 100]
"% of GDP" and similar ratios
non-negative for levels (revenue, spending, debt); can exceed 100 — verify against data before capping. Where a ≤100 bound mostly holds, enforce it with a documented exception set (e.g. PFMH expenditure ≤ 100 outside {Equatorial Guinea, Kuwait, Kiribati} — each with a comment explaining why it's legitimate) so a new country crossing the line fails for review
Balances, growth rates, interest rates
can legitimately be negative — do NOT impose ≥ 0
Mutually exclusive share categories
components sum to 100% (or 1) per (country, year), within a small rounding tolerance (e.g. abs(sum - 100) < 0.1) — first verify against the data that the source's categories are actually exhaustive (some breakdowns ship without an "other" remainder, or only for a subset of rows)
Categorical flags / codes
exact value set (e.g. <= {0, 1}) and non-null
Input schema
set-equality on expected columns — catches the next source rename (PFMH: debt → d) with a clear message
Coverage
country-count floors (≥ previous version); a drop is usually a parsing/mapping regression — re-audit before bumping the constant
Composite sums of optionally-reported components
never a bare .sum(axis=1) — it fabricates zeros on rows where the source reports no component (and hands the whole total to any remainder derived by subtraction). But don't jump to min_count=len(components) either: ground the rule in per-year reporting counts first — when component coverage widens over time (a category only exists from year X), requiring all components can wipe most of the series. The robust pattern is min_count=1 plus a withdrawn-component rule: a component reported by someone last year and by nobody this year marks a partially-published year — null the aggregate there. That rule is release-level by design (it catches a producer publishing without a component table); if one entity's component can plausibly go missing while others still report it, also compare component presence per aggregation key, accepting the false positives genuinely ragged reporting will add. And a component that vanishes for good is a schema retirement — shrink the expected component set rather than treating the year as partially published. (ODA: strict min_count dropped 2453 rows to 453 and erased the US entirely; min_count=1 + withdrawal flagged exactly the preliminary 2025)
Implementation conventions: constants at the top, run() first, check functions belowrun(); sanity_check_inputs(tb) right after loading meadow, sanity_check_outputs(tb) right before paths.create_dataset(...); plain assert with messages that name the offending values. Reference example: imf/2026-06-12/public_finances_modern_history.py.
sanity_check_outputs runs beforetb.format(...), so column names are still as the code produced them — format() lowercases/underscores them afterward. If the step builds columns like f"{col}_{status}_pop" where status is "Legal"/"Illegal"/"missing", the pre-format columns are status_Legal_pop (mixed case), not status_legal_pop. Select such columns case-robustly (e.g. [c for c in tb.columns if c.endswith("_pop")] then filter on c.lower()), or you'll get a KeyError that only surfaces at runtime. Keep update-specific facts (e.g. "country X is Legal in year Y") out of the committed checks — verify those via the garden diff at update time so the checks stay valid across future releases.
Negative-test the checks: after the step passes on real data, simulate each failure mode (rename a column, corrupt a flag, push a value out of bounds) and confirm the matching assertion fires — a check that never trips is untested code.
5c) Country harmonization audit
Run after the garden step completes (and after 5b if it ran). Verifies that the country entities reaching the garden output are canonical, and that the mappings/exclusions consumed by paths.regions.harmonize_names(...) are well-formed. Output: workbench/<short_name>/harmonization_audit.md.
Modern API. Garden steps should be calling paths.regions.harmonize_names(tb, country_col=..., countries_file=..., excluded_countries_file=...) — the wrapper in etl/data_helpers/geo.py:1874. If you find a step still using the deprecated geo.harmonize_countries(...) directly, step 1b's /check-outdated-practices should already have flagged it; treat that as a separate cleanup. The audit below is API-agnostic — both call sites end up emitting the same three warning strings. Some garden steps don't use the harmonizer at all and instead assign country inline in Python (no .countries.json involved); for those, the JSON checks below have nothing to look at — the garden-output check in step 5 is what catches non-canonical entities, so always run it.
Source of truth. Canonical names come from two datasets, both consulted by the harmonizer:
data/garden/regions/2023-01-01/regions — countries, continents, and OWID-defined aggregates. The runtime authority is paths.regions.tb_regions["name"]. This is built from etl/steps/data/garden/regions/2023-01-01/regions.yml plus a merge with regions.codes.csv and field defaults — don't parse the YAML in isolation or you'll miss the legacy entries and produce false positives.
data/garden/wb/<latest>/income_groups — the four World Bank income-group aggregates (High-income countries, Upper-middle-income countries, Lower-middle-income countries, Low-income countries). OWID treats the latest version of this dataset as the official one, so the audit must resolve the version dynamically (don't pin a date — it goes stale when WB publishes a refresh). The names live in the classification column of the income_groups_latest table.
The audit's "canonical" set is the union of these two. A .countries.json entry looks like "Source name": "Target name" — the audit checks that every target name (the value the source gets harmonized to) appears in either dataset. Anything else is flagged.
Scan the log for the three harmonization warnings. These are emitted by etl/data_helpers/geo.py (excluded list) and lib/datautils/owid/datautils/dataframes.py (mapping warnings) — the wording is stable:
rg -n "missing values in mapping\.|unused values in mapping\.|Unknown country names in excluded countries file:" \
workbench/<short_name>/harmonization.log
For each warning, the entity list follows on subsequent lines (because harmonize_countries() is called with show_full_warning=True by default). Capture them.
Validate .countries.json target names against canonical regions + income groups. Each entry maps a source name (key) to a target / harmonized name (value); this check looks at the values. For each garden step in this update:
import json
from pathlib import Path
from owid.catalog import Dataset
# Resolve the canonical regions dataset dynamically (latest built version).
# Don't pin a date — when the regions step version advances, a hard-coded path
# would validate against a stale catalog and flag valid targets as non-canonical.
regions_dirs = sorted(Path("data/garden/regions").glob("*/regions"))
if not regions_dirs:
raise RuntimeError(
"No data/garden/regions/<version>/regions built locally — the audit can't "
"run without the canonical regions catalog. Build it first with "
"`.venv/bin/etlr data://garden/regions/<latest>/regions --private`."
)
tb_regions = Dataset(str(regions_dirs[-1]))["regions"]
canonical_regions = set(tb_regions["name"].dropna().astype(str))
# Add OWID's official income-group aggregates to the canonical set, if available.
# OWID treats the latest income_groups version as official. This artifact is
# often not built locally during a non-income-groups dataset refresh — degrade
# gracefully (warn and skip) rather than aborting the audit.
ig_dirs = sorted(Path("data/garden/wb").glob("*/income_groups"))
if ig_dirs:
ds_ig = Dataset(str(ig_dirs[-1]))
canonical_income = set(ds_ig["income_groups_latest"]["classification"].dropna().astype(str).unique())
else:
print(
"[WARN] No data/garden/wb/<version>/income_groups built locally — "
"skipping income-group enrichment. The four WB income-group aggregates "
"(High/Upper-middle/Lower-middle/Low-income countries) may surface as "
"'not in canonical' until you build that dataset."
)
canonical_income = set()
canonical = canonical_regions | canonical_income
mapping = json.loads(Path("etl/steps/data/garden/<namespace>/<new_version>/<short_name>.countries.json").read_text())
not_in_canonical = sorted({v for v in mapping.values() if v and v not in canonical})
print("Targets not in OWID's canonical regions or income groups:", not_in_canonical)
# Provider regional groupings the source ships (e.g. "Europe (WB)", "Asia and the Pacific (ILO)")
# surface here when OWID hasn't added that provider's regions yet. Flag the "(Provider)"-suffixed ones:
import re
provider_region_candidates = [v for v in not_in_canonical if re.search(r"\([\w .&-]+\)$", str(v))]
print("Look like provider regions not yet in the regions dataset:", provider_region_candidates)
When you report progress to the user during the workflow, never just give a count — always include the list (or grouped categories) so they can judge in one glance.
Checkpoint summary:
"Targets not in OWID's canonical regions or income groups" or "Garden output entities not in OWID's canonical regions or income groups" or "Missing in mapping" non-empty ⇒ stop, decide with user.
"Excluded entries matching canonical regions" non-empty ⇒ stop, ask whether each exclusion is intentional.
"Unused mappings" or "Unknown excluded entries" non-empty ⇒ surface in PR description; not a blocker.
5d) Region-provider drift — if this dataset defines OWID regions, propagate the change
The harmonization audit (5c-3) catches providers not yet in regions.yml. This check is the counterpart: a provider whose aggregates are already in regions.yml can change its regions on a version bump (new/removed/renamed regions, or shifted country membership), and regions.yml won't update itself.
Does this dataset own regions?grep "defined_by: <provider>" etl/steps/data/garden/regions/2023-01-01/regions.yml (e.g. ilo_1/ilo_2, maddison, wid, wb, who). No matches ⇒ skip this step.
Re-extract the provider's region→country mapping from the new version — from the same source the regions were originally derived from (a region/subregion column, region entities, or a table-of-contents tier; see /add-provider-regions Step 1) — and diff it against regions.yml (defined_by: <provider> members), set-equality per region, plus new/removed/renamed regions. If the new version no longer carries the composition, the provider's section in the world-region-map-definitions article links to the defining source — use it to re-derive the provider's existing regions (regions newly introduced by this update won't be documented there yet — get those from the source or ask the user).
If it drifted:stop and decide with the user (composition changes move every region-aggregated value downstream), then update regions.yml and re-propagate via /add-provider-regions — rebuild garden regions, refresh the {provider}_region grapher indicators + metadata, regenerate owid-grapher regions.data.ts (runRegionsUpdater), and update the article section.
6a) Re-evaluate # NOTE: / # TODO: items from step 1c against fresh data
Now that meadow, garden, and grapher have run on the new data, go back to workbench/<short_name>/notes_to_check.md and decide each item's fate. For each entry:
Identify what the workaround does (read the surrounding code).
Load the affected step's output with owid.catalog.Dataset (or inspect the raw snapshot) and compare corrected vs. uncorrected values. Cross-check the producer's release notes / changelog if available.
If the upstream issue is fixed → delete the workaround and its # NOTE: / # TODO: comments in the same commit, then re-run the affected step (use --force --only, add --grapher for grapher) so downstream artifacts pick up the change.
If the workaround is still needed → leave it and add a one-line status under a PR-description section titled "Not covered in this PR" (e.g. "Sierra Leone ×1000 correction still required — raw value in the 2026 file is still ~1/1000 of plausible"). These are deliberately deferred items the next updater should re-check. Delete the whole section if its last item gets resolved mid-PR.
If you're uncertain → keep it, flag it in the PR description, and ask the user.
Do this before step 6b (metadata checks) so any re-runs triggered by comment-removal happen before the metadata sweep, not after.
6b) Metadata quality checks — run after all ETL steps are built
Run all five checks on the newly built garden and grapher datasets so every issue surfaces together. Each skill writes results to the terminal; fix what comes up before moving on.
Typos — /check-metadata-typos scoped to the current step. Run on each of the new .meta.yml files (garden first, then grapher). Accept or skip each suggested fix.
Jinja spacing — /check-metadata-spacing on the built garden and grapher datasets. Catches template artifacts like doubled spaces or stray newlines that only appear after Jinja rendering.
Style guide — /check-metadata-style on the grapher step. Audits user-facing fields (title, subtitle, description_short, display.name, presentation.*) against OWID's Writing and Style Guide. Rules live in .claude/skills/check-metadata-style/STYLE_GUIDE.md, so no Notion access is usually needed — the skill checks the file's Last synced from Notion date and refreshes it from Notion (in a separate PR) when it is more than two months old.
Clarity for a general audience — read every user-facing field with non-specialist eyes. The other three skills enforce structure and style; this one judges whether the text is understandable.
Dimension sweep (dimensional indicators only) — a sentence written for one breakdown renders on all the sibling views, where the view's own filtering can make it false: a caveat that the data doesn't control for X is wrong on the variant grouped by X, a scope word like "all employees" overclaims on a variant filtered to a subgroup, a sentence about a toggle is wrong on views that exist for only one choice of that dimension. Render the text per dimension value (the grapher channel already has it flattened) and read each output as a reader of that chart. Prefer qualifying the wording so it holds everywhere over adding a Jinja branch. Full version: check 6 of the canonical suite in /owid-metadata-generation.
Clarity checklist (do manually, no skill yet)
OWID readers are not domain experts. Walk each indicator's user-facing fields and flag anything that requires inside knowledge to parse.
Field
Clarity check
title / presentation.title_public
A non-specialist should know what the indicator measures from the title alone. Expand acronyms unless universally known (skip GDP; expand GWIS, MFI, SDG, IHME). Don't cram units into the title.
description_short
One or two short sentences: what the metric is and what it covers. No jargon without a gloss. Active voice. The chart subtitle is short by design — no run-on or stacked clauses.
description_key
Free-form markdown prose (paragraphs; sub-lists only where they help — see grapher's descriptionKey-to-string change). Each paragraph or bullet should land a distinct, useful fact. Skip filler ("this dataset is widely used"); prefer substantive caveats (coverage gaps, methodology limits, what counts/doesn't count).
display.name
Short legend label. Reads naturally on a chart axis/legend; doesn't restate the title.
presentation.grapher_config.note
Concise footnote, ≤1 sentence ideally.
Flag and rewrite when you find:
Acronyms or technical terms that aren't expanded the first time they appear
Sentences that only make sense if you already know the data source
Quantitative claims with no unit context (e.g. "burned area" without "in hectares" surfacing somewhere in the user-facing text)
Inconsistent terminology between indicators in the same dataset (e.g. "wildfires" in one, "vegetation fires" in another)
Domain phrases that have a plain-English equivalent (e.g. "anthropogenic emissions" → "human-caused emissions")
Methodology-attribution claims ("following guidance from …") — open the cited link and confirm it actually says that. Agencies revise methodology, so a claim that was defensible when written can be stale now, and the 6c link check only proves the URL resolves, not the claim (real case: metadata cited BEA guidance for deflating data centers with the office PPI; BEA's 2025 annual update had switched them to an industrial+warehouse composite, and the cited page never contained the guidance at all)
Scope qualifiers present in the origin title but absent from user-facing text — if the source is "Private Construction" / adults-only / market-exchange-rate-only, description_short and description_key must say so; the citation line alone doesn't reach readers
Text that repeats what the reader has already read — a description_key bullet restating another bullet, description_short, or the title at the same level of detail. Unpacking description_short in the first bullet (full definition, how it's measured, what's included) is exactly what the panel is for; saying it again in other words is padding. Merge it into the bullet that already covers the ground, or drop it
When a phrasing is ambiguous, propose a concrete rewrite — don't just flag it.
If any skill rewrites a .meta.yml, re-run the affected step so the built catalog reflects the edits. Add --grapher when the affected step is on the grapher channel — without it the local catalog is updated but staging stays stale, so the step 7 indicator upgrade sees the old text.
Then re-run the relevant check to confirm zero remaining violations.
6c) Indicator metadata coverage, dataset block, and link verification
The other quality checks catch content issues; this step catches missing fields and broken URLs before they reach review.
Snapshot DVC freshness.etl update clones the previous snapshot's .dvc content verbatim except for date_accessed. Always re-check date_published and the year in citation_full / attribution under snapshots/<ns>/<new_version>/*.dvc — they will otherwise silently ship the old version's values. Set date_published to the producer's real release date when discoverable; otherwise copy date_accessed. Bump the year in citation_full and attribution to match.
Citation year vs date_published year. After setting both, check whether the year inside citation_full / attribution matches date_published's year. If they differ, confirm it's intentional before shipping: it's legitimate when the producer labels the release by edition rather than publish date (e.g. UN IGME's "2025 report" published 2026-03-17 → citation_full(2025), date_published2026), but otherwise it's a stale citation. When the gap is deliberate, leave a one-line note for the reviewer so they don't re-flag it.
Last-Modified header as date_published source. When the producer's page states no release date (common on fully JS-rendered sites like the IMF Datamapper), the download URL's HTTP Last-Modified header is a defensible source — it's the server's own timestamp for the file, not an inference. Corroborate it against a release-named filename (e.g. …Dec 2025.xlsx + Last-Modified: Fri, 12 Dec 2025) and note the provenance when reporting to the user.
Stale producer description on a JS-rendered page. If the .dvcmeta.origin.description is producer text that may have changed but the page is an SPA shell (static HTML empty, WebFetch 403s, Wayback archives only the shell), don't burn time probing API endpoints and don't rewrite the producer's text to match the data — the blurb can legitimately lag their own releases (FPP shipped 153 countries while the page said 151). Ask the user to paste the page text from their browser, then diff it against the existing .dvc text and apply only the substantive changes. Clipboard pastes flatten typography (curly quotes → straight, en-dashes → hyphens) — keep the existing typographic punctuation unless the words themselves changed.
Mandatory fields per indicator. For every indicator in the garden .meta.yml, confirm these are set (either on definitions.common or per-indicator):
Field
Notes
title
Per-indicator
unit
Common is fine
short_unit
Common is fine
description_short
Per-indicator
description_key
Non-empty (markdown prose, or legacy bullet list); usually common
processing_level
minor or major
presentation.topic_tags
At least one tag
display.numDecimalPlaces
Common is fine
display.tolerance
Common is fine — chart tolerance for missing years
display.name
Per-indicator — required for legend labels
presentation.attribution_short
Set explicitly — does NOT inherit from the origin's attribution_short (verified: MySQL variables.attributionShort stays NULL if it's only on the origin). Place under definitions.common.presentation for the common case.
Conditional: if processing_level: major, every indicator with that level MUST also have description_processing.
Not mandatory (skip if you don't need them): presentation.title_public, presentation.title_variant, presentation.attribution.
Dataset block. Garden .meta.yml MUST include update_period_days:
dataset:
update_period_days: <N>
This controls the auto-update cadence. Even when the rest of the dataset: block is empty, never strip update_period_days — leave the block in place with just that field.
Link verification. Run a HEAD request on every URL in the new .dvc and .meta.yml files (all channels — meadow .meta.yml files matter when they exist). Anything non-2xx is a signal, not a guaranteed break — always double-check before acting:
for url in $(rg --no-filename -No "https?://[^\"' ]+" snapshots/<namespace>/<new_version>/ etl/steps/data/{meadow,garden,grapher}/<namespace>/<new_version>/ \
| sed -E 's/[).,;:>]+$//' \
| sort -u); do
printf "%s %s\n" "$(curl -sI -L -o /dev/null -w '%{http_code}' --max-time 15 -A 'Mozilla/5.0' "$url")" "$url"
done
The --no-filename flag prevents rg from prepending path: to each match (otherwise the for-loop tries to curl path:url and every check returns 000). -A 'Mozilla/5.0' sometimes coaxes a real response out of Cloudflare-fronted hosts, but it doesn't always work — see the next note.
curl non-2xx ≠ broken. Cloudflare-fronted sites (notably ourworldindata.org) can return 404 to curl on URLs that work fine in a browser, depending on edge-node routing, IP geolocation, and cached state. Before treating a 4xx as a real failure:
Re-check with WebFetch (the built-in tool). It uses a different code path and a Mozilla/5.0 UA that Cloudflare usually accepts. A 200 with a coherent page body is authoritative — trust it over curl.
If WebFetch also fails, sanity-check the Wayback Machine. web.archive.org is itself not fetchable by WebFetch — query the availability API with curl instead: curl -sG "https://archive.org/wayback/available" --data-urlencode "url=<url>" (the --data-urlencode matters: interpolating a raw URL truncates the lookup at the first & and can false-clear a broken link). A capture is historical evidence, not a live-link check — read the snapshot timestamp: a capture from the last few days supports the bot-blocking hypothesis; an older capture proves nothing about today (the producer may have retired the URL since). When curl and WebFetch both fail, never clear the link on Wayback evidence alone — report it with the capture date and ask the user to open the URL in their browser (the converse also holds: hosts like BLS block both curl and WebFetch while serving browsers fine, so a double failure doesn't prove the link is dead either).
No automated check is decisive — a suspected-broken link always ends at flag-and-ask. curl, WebFetch, and Wayback only accumulate suspicion: a missing Wayback capture is non-evidence (plenty of live URLs were simply never archived), and a double curl+WebFetch failure can be bot-blocking (see step 2). The only decisive test is the user opening the URL in their browser — so report the evidence trail (statuses, capture date or absence) and ask before rewriting any external link in metadata. Replacing a working link with a "safer" alternative because of a false-positive is worse than leaving the original. Apply the same restraint here as the global "Checkpoints — when to pause" section.
Fix any genuinely-non-2xx hit on url_main, url_download, license.url, or URLs referenced from description / description_key before continuing. The sed strips trailing markdown/punctuation chars (), ., ,, ;, :, >) so URLs inside [text](url) aren't reported as broken because of a stray closing paren.
Anchor fragments — HTTP 200 can't validate them. Servers ignore everything after #, so https://ourworldindata.org/poverty#key-insightsfdd (broken anchor) returns the same 200 as the real #key-insights. For every checked URL that carries a fragment, run a second pass against the page body:
for url in $(rg --no-filename -No "https?://[^\"' ]+" snapshots/<namespace>/<new_version>/ etl/steps/data/{meadow,garden,grapher}/<namespace>/<new_version>/ \
| sed -E 's/[).,;:>]+$//' | sort -u | rg '#'); do
page="${url%%#*}"; frag="${url#*#}"
case "$frag" in *=*|*/*|'!'*) continue;; esac # non-DOM fragments — see below
# grep -F: fragments may contain regex metacharacters (#section.2) — ERE would false-OK id="section-2"
if curl -sL --max-time 20 -A 'Mozilla/5.0' "$page" | grep -qF -e "id=\"$frag\"" -e "id='$frag'" -e "name=\"$frag\"" -e "name='$frag'"; then
printf "OK %s\n" "$url"
else
printf "NO-ANCHOR %s\n" "$url"
fi
done
Skip non-DOM fragments (the case list): anything containing = (scroll-to-text #:~:text=, Google Sheets #gid=…, PDF #page=…) or / (hash-route state, whether slash-prefixed #/dashboard or not — e.g. FAOSTAT's #data/FBS), plus #!… hashbangs — none correspond to an element id and all would false-alarm. Real anchors are heading slugs and never contain = or /.
NO-ANCHOR is a signal, not proof — same epistemics as the curl false-404 note above. Two false-positive modes: the page renders its anchors client-side (the raw HTML lacks the id), or curl got a Cloudflare challenge page instead of the real body. Confirm before flagging: check the fetched body is the real page (grep for </html> and a plausible <title>), then ask WebFetch whether the page has a section/heading matching the de-slugged fragment (key-insights → "Key insights") — OWID heading anchors are slugged headings, so heading-text presence is the authoritative check.
On a confirmed missing anchor the reader still lands on the right page, just at the top — fix the fragment (grep the body's id=" values for the nearest real anchor) or drop it. Apply the same restraint as above: flag and ask before rewriting — the section may exist under a different slug, or the page may not have rebuilt yet.
Verification. After editing, re-run the affected step (with --grapher if grapher) so the catalog reflects the changes. Then confirm presentation.attribution_short actually landed:
from owid.catalog import Dataset
ds = Dataset("data/grapher/<ns>/<v>/<short_name>")
tb = ds["<table>"]
print(tb["<col>"].metadata.presentation.attribution_short) # must NOT be None
Or after the staging upload:
make query SQL="SELECT shortName, attributionShort FROM variables WHERE catalogPath LIKE '%<ns>/<v>/<short_name>%'"
6c-bis) Adversarial data & metadata review (always suggest it — only the run is optional)
You MUST surface this option to the user on every update, without exception. Even when you judge it unnecessary and recommend skipping, still tell the user it exists and let them decide — do not silently omit it. The heaviness of the check governs whether to run it, never whether to mention it.
Offer to run /adversarial-data-review on garden/<namespace>/<new_version>/<short_name>. Step 6c verified that the links resolve; this step actually reads the producer's documentation behind them and cross-checks the data against independent sources online — it's the only step that can catch a mistake made by the source itself (unit slips, wrong-year values, stale pre-revision numbers).
The run is optional because it's the heaviest check in the workflow — fetching methodology docs plus per-value web searches runs ~25–45 web calls and can consume a lot of tokens and time. Skip the run by default; run it when the user opts in, and actively recommend running it when the update shows red flags that only this step can chase down: large unexplained value churn in the diffs, an in-place source revision, a producer new to us, or editorial claims riding on specific values.
Scope for an update: focus the metadata claim review on new/changed text, and the value cross-checks on the newly added data (latest wave/year) plus that skill's standard anchors; deep-review the top-viewed indicators + anomaly-flagged ones per its prioritization step. Apply its routing table: metadata fixes → edit and re-run the step; confirmed source errors → <short_name>.corrections.yml; unconfirmed suspicions → list under "Not covered in this PR" for the reviewer. Save the report path (ai/adversarial-review-<short_name>-<date>.md) in update-context.yml and summarize any 🔴/🟡 findings in the PR body. Run this before 6d/commit so the fixes land in this PR.
6d) Scheduled-issue workflow check (owid-issues)
Every recurring data update is driven by a scheduled GitHub Actions workflow in the owid/owid-issues repo (.github/workflows/update-*.yml) that periodically opens a "Data update" issue. The conventions live in the Notion page "Scheduled data issues": schedule anything updated at least once per year but less than daily; filename update-{namespace}-{short_name}.yml; a cron schedule: trigger + imjohnbo/issue-bot creating the issue. This step runs now because 6c just established the two cadence facts the cron must match — dataset.update_period_days and the producer's actual release rhythm (source.release_date / next_release in update-context.yml).
Locate the workflow. The filename convention is loosely followed in practice, so search in widening circles — a miss on the exact name proves nothing:
Use the local checkout ~/owid-issues if present (git -C ~/owid-issues pull first); otherwise clone it (gh repo clone owid/owid-issues ~/owid-issues). Don't fall back to a gh api …/contents filename listing — the checks below need file contents (content grep for group workflows, cron/body/assignees parsing), and the commit step needs a working tree anyway.
Exact conventional name update-<namespace>-<short_name>.yml → fuzzy filename match (hyphen/underscore swaps, dataset-title words — e.g. update-gallup-ai-indicator.yml covers gallup/ai_indicator) → content grep (rg -il "<short_name>|<namespace>|<title words>" ~/owid-issues/.github/workflows/).
Group workflows count. One workflow may cover a family of related datasets (e.g. update-climate.yml → /climate-update; the quarterly update-war-ucdp-preview-q*.yml set; an "… + OMM" title covering a derived chain). If a group workflow covers this dataset, verify that workflow — don't create a per-dataset duplicate.
If found, verify three things:
Frequency + timing. Parse the cron: line. The implied period must be consistent with update_period_daysand with the release cadence observed this update. Check the timing within the cycle too: the issue should fire shortly after the producer's expected publication window, never before (existing precedents: Gallup 0 8 15 */3 * — mid-month, just after the wave publishes; OECD health expenditure 0 0 8 7 * — right after the early-July release). If this update revealed that the cadence or window shifted, propose a new cron with a # comment in the YAML explaining the timing choice (matching the existing style) — cron changes need user sign-off before committing.
Naming. The filename should be update-{namespace}-{short_name}.yml. Deviations → flag in the report only; don't rename (churn, and behavior doesn't depend on the filename). Exception: a file missing its .yml/.yaml extension is genuinely broken — GitHub Actions silently ignores it — fix that without asking.
Actionable issue body. The body should name the dataset (no version — versions go stale) and tell the next updater to activate the Claude skill:
title: "Data update: <dataset title>"
body: |
Update the <dataset title> dataset (`<namespace>/<short_name>`).
To run the update with Claude Code, run:
```
/update-dataset <short_name>
```
For group workflows, keep it a single command listing every member dataset — /update-dataset <short_name1> <short_name2> — or point at the family skill (e.g. /climate-update). If the body lacks the /update-dataset pointer or references a renamed path, refresh it — body/title fixes are auto-applied and reported afterwards, no need to ask. Also check assignees: still points at the dataset's current owner; flag a mismatch, don't auto-change.
If not found: per the Notion rule, any dataset with update_period_days roughly in [2, 366] should have a scheduled issue (err on the side of scheduling too much). Propose creating one: copy an existing workflow as template (imjohnbo/issue-bot@v3.3.6 shape; keep close-previous: false and its WARNING comment), cron shortly after the expected release window, assignees: = the GitHub handle of the human directing this update (team table in CLAUDE.md), filename per the convention, title/body per the template above. If this update touched several related datasets, propose one grouped workflow rather than several. Creating a new workflow needs user sign-off.
This SKILL.md is very large, so SkillsMP previews the first section here.View on GitHub
regions.yml
defined_by: <provider>
regions.yml
/add-provider-regions
Grapher step: run + verify (skip diffs), or explicitly mark N/A
Re-evaluate each catalogued # NOTE: / # TODO: against fresh data; delete resolved workarounds + comments together, or record status in PR body
Verify indicator-metadata coverage, dataset.update_period_days, snapshot DVC date_published and citation_full year (etl update copies both verbatim — bump to the producer's real release date / year, or to date_accessed / current year if the source doesn't publish one), and that all URLs resolve (HEAD-check) and every #fragment matches a real anchor in the target page (anchor pass, see 6c)
Always suggest the optional adversarial data & metadata review (/adversarial-data-review) — verify metadata claims against the producer's fetched documentation and cross-check values against independent sources. Surfacing this offer to the user is mandatory every run (even when you recommend skipping it); only the run is opt-in — it's heavy (~25–45 web calls), so skip by default and run on user opt-in or visible red flags (see 6c-bis)
Scheduled-issue workflow check (owid-issues): locate the dataset's update-*.yml (exact / fuzzy / group match), verify cron vs the observed release cadence + update_period_days, filename convention, and that the issue body says to run /update-dataset <short_name>; auto-fix body/title, ask before cron changes or new workflows — commits go straight to owid-issues main (see 6d)
Commit, push, and update PR description
Run indicator upgrade on staging and persist report
Run the hardcoded-time-bounds audit (check-hardcoded-years) after all remaps — numeric minTime/maxTime/timelineMin/MaxTime/map.time pins on every surface carrying the new indicators (charts, MDim/explorer views, narrative charts, article time= links), graded against the new data's latest time; a pin below it means the update is invisible on that surface — propose "latest" fixes with user sign-off (see step 7)
Always suggest the optional empty-entity audit (check-empty-entities) after all remaps — it sweeps every surface (charts, map columnSlug, MDim views, explorer views, narrative charts, gdoc country= references). Surfacing this offer to the user is mandatory every run (even when you recommend skipping it); only the run is opt-in — it can consume many tokens on widely-charted datasets, so run on user opt-in or when the remap touched many views; regressions vs production = fix, pre-existing = document (see step 7)
Update update-context.yml with published chart count and 1–3 chart views for the public announcement
Render Slack announcement via data-updates-comms, save to workbench, post @codex review as a separate PR comment, and notify user to post it to #data-updates-comms
Draft the public-facing "Data update" post for OWID /latest via data-update-announcement (Mode A) — two versions, user picks, then the Google Doc in /Data updates, and hand the user the link (not added to the PR). The skill declines when we posted about this data less than six months ago; a declined post is a completed item — record the eligible date, don't override
Run downstream-dependency check (rg "<namespace>/<old_version>/<short_name>" dag/ -g "*.yml" | grep -v "^dag/archive"); for each consumer outside the dataset's own chain, decide with the user whether to bump in this PR or document under "Downstream dependencies" for a follow-up PR (see "Downstream dependency check" section below for details)
Run the silent-breakage check whenever downstream consumers were repointed in this PR: confirm the buildkite/etl-automated-staging-environment PR check is green (red = a consumer crashed on staging, and the report under-reports until it's fixed; .venv/bin/etlr --modified --continue-on-failure --private is the optional local equivalent for small fan-outs), then triage the data-diff report — every red "− lost N data point(s)" entry in its Top-changes list and every 🔴-tier dataset (see "Silent-breakage check" section) and run the full-report audit probes (structural / World / raw-country / >30% / wipe-vs-edge per loss)
Ask the user whether to remove the old DAG entries; if yes, delete them and their files AND relocate the new entries into the old slot (see "Removing the old version & reordering the DAG") — don't forget this step
Hand off the QA links to the user (Anomalist + Chart Diff on the staging branch, plus the data-diff report) — this is the final step
After running, always verify the dag file: grep for the old version and confirm all internal references between the new steps point to the new version (e.g., garden depends on new meadow, not old meadow).
etl update writes the new entries in the flat DAG form — convert them to the nested (compact) form now, while you're in the file, rather than leaving it until archiving (otherwise the flat block tends to survive the whole update unnoticed). See the example and load_dag() parse-check under "DAG archiving & reordering" step 4.
Run, fix, re-run; produce diffs
Save diffs and summaries
Diff against a freshly-rebuilt old version, not the stale feather on disk. The old version's data/garden/.../ feather was built whenever it last ran — possibly against an earlier snapshot of a shared upstream dataset (population, regions, income_groups). A fresh build of the new version uses the current upstream, so a naive new-vs-old-feather diff shows differences in every population-weighted cell across all years and regions — pure upstream drift that has nothing to do with your change. Before trusting any diff, rebuild the old version on the current catalog and diff against that:
The apples-to-apples diff should collapse to just your intended change. Mention the drift separately in the PR (Chart Diff on staging will show it, because the live data is also stale relative to current upstream). This bit me twice in one update — don't skip it.
When NaN can appear on one side, don't let .fillna(False) hide it. In a cell-by-cell diff, (a - b).abs() <= tol evaluates to NaN when exactly one side is NaN; a downstream .fillna(False) then silently drops that real one-sided change. Treat "one side NaN, other side not" as a difference explicitly.
Spot-check derived diff columns before quoting them. A pct-change column built via an assign/sort_values chain on merged Tables can silently misalign against its own rows. Before a revision magnitude goes into the PR body or an announcement, recompute it in a fresh plain-pandas frame straight from the raw old/new files and eyeball one row (old value, new value, pct) for arithmetic consistency — a mismatch on a single row means the whole column is garbage. (This produced "median +8.2%" where direct recomputation gave +14.3% in a real update; the wrong number was already in the PR body before an adversarial re-check caught it.)
Surface new indicators. Collect columns present in the new version but absent from the old — read both the meadow diff (step 4) and the garden diff, because the pipeline may drop columns between the two. etl diff prints additions as green + Column lines, but the ranked HTML report scores them at severity 0 — they sort last and will not surface on their own; read the text/JSON output, or compare column sets directly (set(ds_new[t].columns) - set(ds_old[t].columns) per table). Classify each find:
New in meadow and in garden → genuinely new indicator: warn the user and list it in the PR body under a "New indicators" heading — it appears on no chart until someone curates it, needs the same metadata coverage as the rest (see 6c), and may deserve a mention in the update announcement (step 9).
New in meadow but missing from garden → the pipeline drops it (explicit column selection or .drop(columns=...) in garden): surface it as "the source now ships X — currently dropped" and let the user decide deliberately whether to keep it.
Paired + Column / − Column → usually a rename, not an addition — route it to the indicator-upgrade mapping (step 7) instead.
Long-format tables hide additions from the column diff. When a table keys its series by a dimension column (indicator, metric, question, …) with a stable value column, a new series adds rows, not columns — the column-set diff stays empty. Also diff the distinct combinations of the identifying dimensions, not each dimension's values separately — a new series can be a new tuple of existing values (e.g. sex=male and metric=count both existed before, but the (male, count) series is new): set(tb_new.reset_index()[dims].itertuples(index=False, name=None)) - set(tb_old.reset_index()[dims].itertuples(index=False, name=None)) where dims is the list of identifying dimension columns — built tables keep them in the index after Table.format(...), hence the reset_index() (with a single dimension this reduces to a value-set diff). The grapher step expands dimension values into separate variables, so the step-7 shortName pairing surfaces these too — but catch them here, before staging.
New companion files at the source are new-indicator candidates too. Every check above diffs within the file we snapshot — a producer adding a separate file to the release (a pre-built index, a summary panel, a codebook-derived extract) is invisible to all of them. When listing the source's files (release page, OSF/Zenodo file API), diff the file inventory against the previous cycle — the baseline is the previous snapshot's .dvc companion-files # NOTE: when the host has no file-history API (/create-snapshot step 6 writes it; refresh it in the new .dvc either way) and treat each new data file as a candidate: ingest it (usually a second snapshot + its own table in the same steps), or record the deliberate skip in the PR body. (Velasco LGBTI: the revision added a composite-index CSV — the successor of a retired dataset that a live chart still ran on — and the column diff could never see it.)
If the meadow step itself selects a hardcoded column subset (tb[[...]], usecols=, .drop(columns=...)), a new source column never reaches any diff — grep the meadow step for such selections and, when present, compare the raw snapshot column set old-vs-new (snap.read_*().columns) to catch additions dropped at the door.
Deflated ("real"/"constant") series
the deflator must cover every period of the nominal series — a how="left" merge silently yields a NaN tail that ends the real series early when the deflator lags; also assert the base-period row (prefer the producer's own annual-average row, e.g. BLS's M13) exists
A non-empty not_in_canonical list means the mapping points at entities that aren't registered in either the regions catalog or the income-groups dataset. This isn't automatically a bug — it's a heads-up. Stop and decide with the user before proceeding — same pattern as the global "Checkpoints — when to pause" section at the top of this skill. Common causes (in order from "fix" to "accept"): typo, retired alias used as canonical, casing/whitespace mismatch, or a custom aggregate the source defines that OWID has no equivalent for. For typos/casing — fix the JSON.
If the unmatched targets are a provider's regional grouping that OWID doesn't define yet — the provider_region_candidates above ("(Provider)"-suffixed names like "Europe (WB)", "Sub-Saharan Africa (ILO)"), or the source ships a region/subregion column feeding these — don't just accept them as outside-the-system. The proper fix is to add that provider's regions to the regions dataset with the /add-provider-regions skill (separate PR): they then become canonical, merge with regions/population infrastructure, and get a {provider}_region map indicator, instead of living outside the system. Surface this to the user and offer to run /add-provider-regions. Only for genuinely one-off, non-geographic groupings the regions system shouldn't own (BRICS, G7, G20) — accept and note in the PR description that those live outside the canonical system. For a real new historical region — add an entry to regions.yml in a separate PR.
Audit .excluded_countries.json. The file is optional; skip if it doesn't exist:
excluded_path = Path("etl/steps/data/garden/<namespace>/<new_version>/<short_name>.excluded_countries.json")
if excluded_path.exists():
excluded = json.loads(excluded_path.read_text())
suspicious_canonical = sorted(set(excluded) & canonical)
# Also surface continents and aggregates separately for review
aggregates = set(tb_regions[tb_regions["region_type"].isin(["continent", "aggregate"])]["name"].dropna().astype(str))
suspicious_aggregates = sorted(set(excluded) & aggregates)
print("Excluded entries that ARE canonical regions:", suspicious_canonical)
print("Excluded entries that are continents/aggregates:", suspicious_aggregates)
print("Full excluded list for review:", sorted(excluded))
suspicious_canonical is the actionable signal: each entry is a known country/region that we are dropping. Sometimes this is intentional (e.g. dropping "World" rows because the source double-counts them) — surface, don't auto-fix. Pause and ask the user if the list is non-empty. The full list is dumped so the LLM can also eyeball it for entities that aren't in canonical but look like real countries (typos, alternative names) we should be mapping rather than dropping.
Audit garden output entities. Always run this check, regardless of whether .countries.json exists or is populated — JSON mappings describe inputs to the harmonizer, but the entities that actually reach Grapher are whatever sits in the country column/index of the built garden tables. Inline country assignments (e.g. hardcoded tb["country"] = "England and Wales") and post-harmonization mutations both bypass the JSON check entirely; this is the only step that catches them.
from pathlib import Path
from owid.catalog import Dataset
garden_dir = Path("data/garden/<namespace>/<new_version>/<short_name>")
ds_garden = Dataset(str(garden_dir))
entities: set[str] = set()
for tname in ds_garden.table_names:
tb = ds_garden[tname]
# `country` can live in the index (after .format()) or as a regular column.
if "country" in tb.index.names:
entities.update(tb.index.get_level_values("country").dropna().astype(str).unique())
elif "country" in tb.columns:
entities.update(tb["country"].dropna().astype(str).unique())
# tables with no country column are silently skipped (e.g. reference tables)
output_not_in_canonical = sorted(entities - canonical)
print("Garden output entities not in OWID's canonical regions or income groups:",
output_not_in_canonical)
Same triage rules as the JSON-targets check (Python check #3): typo / casing / alias / legitimately custom aggregate. A non-empty list means at least one entity that ships to Grapher isn't registered in either the regions catalog or the income-groups dataset. Stop and decide with the user before proceeding. Common fixes: typo or casing → patch the inline assignment (or .countries.json, whichever is the source) so the value matches the canonical name; alias → switch to the canonical name; legitimate custom aggregate → accept and note in the PR description that the entity lives outside the canonical system.
Write findings to workbench/<short_name>/harmonization_audit.md with six sections, populated only when non-empty. Each section must list every flagged entity, not just a count — counts alone aren't actionable, the user (or you) needs to read the actual names to judge whether each is intentional. For long lists (>20 entries) group by pattern when the grouping is obvious (e.g. ILO's " (ILO)"-suffixed regions vs. international orgs vs. derived "World ..." aggregates) so the reviewer can scan categories instead of one flat list. Sections:
## Missing in mapping — countries in source data not in .countries.json (from log warning #1) — list each missing source name
## Unused mappings — .countries.json entries the data never used (warning #2) — list each unused source→target pair
## Unknown excluded entries — .excluded_countries.json entries not present in source data (warning #3) — list each
## Targets not in OWID's canonical regions or income groups — target names from .countries.json that aren't registered in either dataset (Python check #3) — list each target name and the source names that map to it
## Excluded entries matching canonical regions — possible over-exclusion (Python check #4) — list each
## Garden output entities not in OWID's canonical regions or income groups — distinct country values found in the built garden tables that aren't in canonical regions or income groups (Python check #5) — list each entity
Surface in PR. If any section was populated, add a collapsed "Harmonization audit" section to the PR description (after the per-step sections) with the same listings, not just a summary. Empty sections can be omitted.
