| name | dataset-themes |
| description | Group the dataset catalog into collections by theme and by geography, and keep theme tagging honest. Use whenever the user wants to find datasets by theme or by country/global, see what's available in a topic area (finance, energy, transport, land use…), regenerate the by-theme or by-geography collections, reconcile or clean up theme tags, or check which review folders are missing from the catalog. Writes dataset-review/collections/by-theme.yaml and by-geography.yaml; reads the catalog but never edits it. Complements dataset-review, which produces the individual entries this skill groups.
|
Dataset themes (group & reconcile)
Turn the catalog into facet collections: "what datasets do we have on X?" and
"what do we have for Chile / globally?", answered from one source of truth
rather than from memory or folder-spelunking. The catalog
(dataset-review/catalog/index.yaml) already carries a themes: list and a
coverage block per dataset — this skill normalises the themes through a
controlled vocabulary and materialises two generated facet files alongside the
hand-curated collection.yaml: by-theme.yaml (one collection per theme) and
by-geography.yaml (one per country, one global, one per region).
The grouping is mechanical; the judgment is in the vocabulary. The agent drafts
and runs; the human owns what the canonical themes are and which uncataloged
datasets get added. Don't invent a parallel theme store — index.yaml stays
the single source of truth, and references/themes.yaml is the lens over it.
What lives where
The controlled vocabulary is references/themes.yaml: canonical theme ids, each
with a definition and the legacy/raw tags (aliases) folded into it, plus a
qualifiers list for tags that describe a dataset rather than its subject
(method, quality, framing) and so must not create theme buckets. The grouping
logic is references/group_by_theme.py — read-only over the catalog, writing
only the generated by-theme.yaml.
Three files live in dataset-review/collections/. collection.yaml is
hand-curated and the skill never touches it — it is the home for judgment-based
bundles that aren't a single facet (a need's working set, a pilot bundle).
by-theme.yaml and by-geography.yaml are generated by this skill and must not
be hand-edited. Generated ids are prefixed and fully hyphenated to stay distinct
from each other and from curated ids: theme ids become theme-<hyphenated> (the
canonical theme's underscores become hyphens, so climate_finance yields
theme-climate-finance); geography ids become geo-chile, geo-global,
geo-<region-slug>.
Because theme and geography are now both auto-derived, anything in
collection.yaml that is just a country or a single theme is redundant with the
generated files; keep collection.yaml only for genuinely curated cross-cuts.
Actions
Generate — group datasets into facet collections
Run python references/group_by_theme.py from any cwd (paths resolve from the
script location). One run rewrites both facet files and prints the coverage
health-check to stderr: by-theme.yaml (one theme-<hyphenated-id> collection
per canonical theme) and by-geography.yaml (one geo-<country> per country,
geo-global, and geo-<region-slug> per region). Use --no-write to preview
without writing, and --report to also print the markdown by-theme index. The
files are regenerated, never hand-edited; if a theme grouping looks wrong, fix
the tag in index.yaml or the alias in themes.yaml, and if a geography
grouping looks wrong, fix the coverage block in index.yaml, then re-run.
Each dataset appears under every theme it carries (datasets are legitimately
multi-theme) but lands in exactly one geography bucket (by country if countries
are listed, else global, else regional). A dataset with no recognised theme — or
no geography — is listed as a commented-out reminder at the foot of the relevant
file; a theme tag that is neither canonical, alias, nor qualifier shows up under
Unknown tags in --report. All are work-to-do signals, not errors to hide.
Keep in sync — --check
The generated files go stale the moment index.yaml changes without a re-run, and
a stale view misleads silently. python …/group_by_theme.py --check regenerates in
memory and exits non-zero if the committed files differ — run it (or rely on the
repo's pre-commit hook in .pre-commit-config.yaml, which calls it) after editing
the catalog. The same command suits CI. When in doubt about freshness, regenerate
before trusting a grouping.
Reconcile vocabulary — when tags drift
Unknown tags mean the vocabulary and the catalog disagree. Resolve each by
either (a) adding the tag as an alias of an existing canonical theme, (b)
promoting it to a new canonical theme when it's genuinely distinct and will
recur, or (c) listing it under qualifiers if it's a characteristic, not a
subject. Then fix the offending themes: entry in index.yaml to use the
canonical id so future reads are clean. Present proposed vocabulary changes for
the human to approve before writing them — the canonical set is theirs to own.
Keep the vocabulary lumpy on purpose: a theme that only ever has one dataset is
a bucket of one, and usually wants merging into a broader parent. Split only
when the distinction is load-bearing for how datasets are actually searched.
Coverage / backfill — keep the catalog complete
The coverage health-check (printed to stderr, or via --coverage-only) diffs
review folders on disk against the folder each catalog entry points at (its
release path). "On disk, missing from catalog" are datasets invisible to theme
grouping until added — for each, read its README.md, propose a catalog entry
(id, name, publisher, coverage, and canonical themes:) drawn from the review,
and let the human confirm before appending to index.yaml. "In catalog, no
matching folder" is a stale path or a not-yet-committed review — reconcile the
path or add the folder. Adding catalog entries is the dataset-review skill's
Step 6 mechanic; defer to it for the entry format rather than restating it here.
Hard rules
index.yaml is the single source of truth for themes; this skill reads it and
writes only the generated collections/by-theme.yaml. Catalog or vocabulary
changes need human approval; never fabricate a theme tag for a dataset without
evidence from its review.by-theme.yaml and by-geography.yaml are generated output; collection.yaml
is hand-curated. Do not hand-edit the generated files and never let the skill
write collection.yaml; to change a grouping, fix the themes:/coverage in
index.yaml (or the vocabulary) and regenerate.- Qualifiers are not themes. Acquisition method, data quality, and analytical
framing describe the data, not its subject; keep them out of
themes: so
grouping stays meaningful.
- One dataset can hold several themes — group by membership, never force a
single primary theme.
- Markdown formatting: do NOT hard-wrap prose at a fixed column width; write
each paragraph and bullet as one long line and let the editor soft-wrap.
- Prose style: follow
knowledge-base/topics/writing-style.md — lead with
plain-language prose before any list, impersonal voice, identifiers collected
in references rather than sprinkled through prose.
Adapting to catalog changes
The index structure is expected to evolve (themes may even move out of
index.yaml). Everything the script assumes about where fields live is
isolated in three accessors at the top of group_by_theme.py — raw_themes,
coverage, release_path. If the catalog is restructured, repoint those and
the rest of the script keeps working; if themes move to a separate file, only
raw_themes changes (load the new source there).
References
references/themes.yaml — controlled theme vocabulary (canonical ids,
aliases, qualifiers). DRAFT until adjudicated; this is the human-owned piece.dataset-review/collections/README.md — orientation for the collections folder..pre-commit-config.yaml — optional hook running --check on catalog edits.references/group_by_theme.py — generates by-theme.yaml and
by-geography.yaml; prints coverage.dataset-review/catalog/index.yaml — the catalog this skill reads.dataset-review/collections/by-theme.yaml — GENERATED theme collections (this
skill's output; do not hand-edit).dataset-review/collections/by-geography.yaml — GENERATED geography
collections (country / global / region; do not hand-edit).dataset-review/collections/collection.yaml — hand-curated cross-cut
collections; the skill never writes this file.
