CityCatalyst-global-data

Manufacture a structured staging dataset by compiling many public sources (award lists, registries, reports, portals) into schema'd, provenance-flagged, verified records — for cases where no single external dataset exists to adopt. Use whenever the user wants to compile, harvest, scrape, assemble, or "go collect examples/records" from scattered public sources; build a dataset that doesn't exist yet; or turn many announcements into one queryable table — even if they never say "compile". Sibling to dataset-discovery: discovery *finds* an existing dataset, compile *builds* one. Writes only to dataset-compile/<slug>/, never the knowledge base or the catalog.

Build the Mage pipeline that ships a vetted dataset into the modelled database — the third stage after discover → review. Use whenever the user wants to build/create a pipeline, ship or promote a reviewed dataset to production, load a cleaned dataset into modelled.*, register a dataset in the catalog, upload source/cleaned data to S3, or "wire up the pipeline for <dataset>". Orchestrates the stage and defers the how-to to knowledge-base/topics/engineering-standards/; it does not restate those standards.

Validate and profile a dataset after a pipeline has loaded it into the modelled database. Use when the user wants to test/validate loaded data, check a modelled table, write or run data expectations, check nulls / value sets / ranges, produce a data-quality or validation report, or "say something about the data". The post-load layer that complements Mage's in-pipeline @test checks; runs against the live DB.

Deep-dive review of a dataset for CityCatalyst: verify access and license, profile the actual data, read the methodology, run the fit-for-purpose check, and draft the review entry in dataset-review/reviews/. Use whenever the user wants to review, assess, vet, or "go deeper into" a dataset; promote a discovery candidate; check whether a dataset is fit for purpose; or create/update an entry under dataset-review/ — even if they just say "let's look at this dataset properly". Follows dataset-discovery in the pipeline: discover → review → pipeline.

Group the dataset catalog into collections by theme and by geography, and keep theme tagging honest. Use whenever the user wants to find datasets by theme or by country/global, see what's available in a topic area (finance, energy, transport, land use…), regenerate the by-theme or by-geography collections, reconcile or clean up theme tags, or check which review folders are missing from the catalog. Writes dataset-review/collections/by-theme.yaml and by-geography.yaml; reads the catalog but never edits it. Complements dataset-review, which produces the individual entries this skill groups.

Find and triage candidate datasets for CityCatalyst data needs. Use whenever the user wants to find, source, scout, or search for data; mentions a data gap; asks what data exists for a sector, theme, or geography; or wants to open, update, or resolve a data need — even if they never say "search" or "discovery". Also use when deciding whether a dataset someone suggested is worth a full review. Produces a need folder in dataset-discovery/needs/ containing need.md, search.yaml, and candidates.yaml.