dataset-review

name

dataset-review

description

Deep-dive review of a dataset for CityCatalyst: verify access and license, profile the actual data, read the methodology, run the fit-for-purpose check, and draft the review entry in dataset-review/reviews/. Use whenever the user wants to review, assess, vet, or "go deeper into" a dataset; promote a discovery candidate; check whether a dataset is fit for purpose; or create/update an entry under dataset-review/ — even if they just say "let's look at this dataset properly". Follows dataset-discovery in the pipeline: discover → review → pipeline.

Dataset Review (deep-dive)

Take a dataset — usually a promote candidate from dataset-discovery/ — and turn claims into verified findings: what's actually in the data, what the methodology means for our use, what the license really says, and whether it's fit for purpose. Output is a review entry in dataset-review/reviews/<publisher>/<dataset>/ in the house style.

The work is exploratory; the deliverable is not. Don't script the path — contract the destination: by the end, every question in references/deep-dive-checklist.md has an answer tagged verified, inferred, or unanswered. Never present an inferred or skimmed answer as verified — confident summaries of unread docs are this skill's main failure mode, and the tags are the defense.

Checkpoints — work in increments

Run as a sequence of steps, stopping after each one to present findings and let the user decide to continue. The human owns judgment calls (fit verdict, what goes in warnings, promotion); the skill owns legwork (fetching, profiling, cross-checking). If the user explicitly asks for an end-to-end run, do all steps but still present per-step findings in the final report.

Step 1 — Provenance, access, license

Fetch the authoritative metadata record (publisher page, DOI record, data catalog entry), not blogs about it. Establish: canonical download(s) and versions, license as stated in actual terms (a landing-page "open" claim is inferred, not verified), citation requirements, update cadence or frozen status. If multiple published versions exist, identify which is canonical for us and why.

Files often can't be fetched directly from this environment — asking the user to download and drop the file in is a normal step, not a failure.

Step 2 — Profile the data

Open the real file. Establish structure (sheets/tables/layers), actual columns and units, the real row count vs claimed coverage, and — most valuable — the quirks a pipeline would trip on: working-notes rows, phantom padding, trailing spaces, inline section headers, truncated labels, sentinel values, files that are exports of office tools rather than data products. Use the data plugin's explore/profile skills where available. Every quirk found here is a parsing note in the README.

Step 3 — Methodology

Read the methodology documentation (figure captions, methods sections, technical annexes — primary sources). Establish: what the values actually measure, against what baseline, with what uncertainty, and what the authors explicitly exclude or warn about. Translate each into a concrete implication for our use case — "non-additive" matters differently for ranking than for inventory. Author-stated caveats are gold: collect them.

Step 4 — Fit-for-purpose check

Score against the originating need's search.yaml (or elicit criteria if the review wasn't triggered by a discovery need): each must-have, constraint, and the screening guidance, each marked pass/fail/conditional with evidence. By now this should be nearly mechanical — if it requires fresh research, steps 1–3 missed something.

Also check the catalog for sibling datasets by theme — complements and overlaps (e.g. a feasibility dataset pairing with a potentials dataset). The fastest way is the generated theme view collections/by-theme.yaml (and by-geography.yaml for country/global siblings); fall back to searching index.yaml by theme keywords. Look across publishers, not just the one you expect — adjacent datasets under other publishers are easy to miss.

Step 5 — Write the review entry

Create reviews/<publisher>/<dataset>/README.md (house style and section guide in references/readme-template.md; good live examples: reviews/ipcc/ipcc-ar6-spm7-mitigation-potentials/, reviews/ipcc/ipcc-sr15-mitigation-feasibility/) and place files in releases/<version>/. Version naming: the publisher's version or release year/date — match the publisher's own framing.

File placement convention inside a release: data/ is committed to git and holds only what helps someone understand the dataset later (cleaned outputs, mappings, and the canonical raw when genuinely small — single-digit MB); aggregate charts are rendered inline in the notebook only (clean and readable) — never call fig.savefig(...), never write .png/image files, and never create a figures/ folder. The rendered notebook is the single home for the visuals; review.md gives the precise aggregates as tables and points to the notebook for the charts; sample/ is gitignored (see repo .gitignore) and holds large files and working outputs. Large raws go in sample/ with re-download instructions (URL/DOI + retrieval steps) in the README so the notebook can be re-run from a fresh clone. Notebooks sit at the release root.

The README's most valuable sections are interpretation warnings (what silently bites downstream use) and parsing notes (what breaks naive ingestion) — write both for the engineer and analyst who will never read the methodology themselves.

Phase B — Extraction notebook (after the README exists)

Turn the README's parsing notes into working, validated code. Method and block-by-block skeleton in references/notebook-template.md. The core discipline: parsing notes ↔ cleaning blocks stay 1:1 — if cleaning needs a step the README doesn't mention, add the parsing note first; if the notebook discovers a new data fact (e.g. source-side rounding residues), it flows back into the README. Fit analysis is limited to the questions the need raises — that limit is what keeps notebooks from sprawling.

Phase C — Mapping (when the need requires a taxonomy/schema link)

Produce a crosswalk csv in the release data/ linking the dataset's units to the target taxonomy (TEF, GPC, locodes), in the house mapping schema. Method in references/mapping-method.md. Core principles: link, don't attribute (values stay at source granularity — never split a quantity across mapped targets); keep unmapped entries with mapping_confidence: none and a rationale; agent drafts with confidence tags, human adjudicates the mediums and lows; record the adjudication date and adjudicator in the notebook findings.

Phase D — review.md (close out the release)

Write the release's epistemic contract: what the data can and cannot support, as literal claim/anti-claim sentences with conditions. Template in references/review-template.md. This is the document downstream analysts read instead of the methodology — most of its content falls out of phases A–C (author caveats → anti-claims, fit analysis → claim conditions, mapping coverage → downstream rules), so write it last, when those are known. It is then a living document within the release: whenever a new artifact (another mapping, a new analysis cell) changes what the data can support, update review.md in the same change.

Phase split criterion (for future maintainers): a phase becomes its own skill only when it's routinely run without the others AND its reference file outgrows this skill. Until then, one skill keeps the README ↔ notebook ↔ mapping ↔ review.md loop in one place.

Step 6 — Promotion mechanics (on user approval)

Add/update the catalog/index.yaml entry; if the review came from a discovery need, collapse the candidate to a promoted stub and update the need's outcome. Never delete old release folders — new releases sit alongside old ones; production_approved in the catalog decides what's active.

Hard rules

• Evidence tags are mandatory: verified / inferred / unanswered. License and methodology claims must be verified from primary sources before the README states them plainly.
• dataset-review/ follows publisher/dataset/version structure strictly. Keep markdown minimal: per review folder, the only markdown is README.md and the release review.md — plus, where a review acts as the hub for a cross-cutting working methodology, a single methodology.md at that review's root. No other ad-hoc markdown. Working/WIP methodology lives in that methodology.md (clearly marked working), NOT in knowledge-base/topics/, which is reserved for landed reference; promote a consolidated version to topics only once the method stabilises.
• The review records what the data is, including flaws — fit problems go in warnings, not silently dropped. A negative review (not fit) is a valid outcome; record it rather than abandoning the entry.
• Markdown formatting: do NOT hard-wrap prose at a fixed column width. Write each paragraph and each bullet as a single long line and let the editor soft-wrap. (This bullet is one line — follow its example. Code blocks and tables keep their natural structure; existing wrapped files can be left alone or unwrapped when touched.)
• Engineering style: notebook and pipeline code follows knowledge-base/topics/engineering-standards/ (block flow, idempotency, parameters, the three validation layers, naming, definition-of-done) and the notebook shape in references/notebook-template.md. Defer to those docs rather than restating them. When a review surfaces a better engineering pattern, promote it into the relevant engineering-standards/ doc so it becomes enforceable rather than tribal, instead of leaving it to live only in a worked example.
• Open a review folder only when the source is applicable AND profilable now. Two gates, both checked at Step 1: (a) applicable to our geography/use (not an ODA-only facility for a DAC-graduated country, not a fund that excludes our actor universe, actually covers our region); (b) profilable now — there is real content to review (an operational call, or a national portfolio/dataset), not just an eligible-in-principle programme that has no live instance yet. If either gate fails, record the finding in the originating dataset-discovery need (deprioritize/reject for not-applicable; investigate + "promote when live" for applicable-but-not-yet-profilable) — do not create a dataset-review/reviews/ folder. A review folder is for a source worth profiling today; everything else lives in discovery so reviews don't fill with dead or empty entries. (If a folder was already opened before a gate was found, delete it and move the finding back to discovery.)
• Concision / cross-references: keep entries tight. Reference another review, the oef inventory, or a need only when it tells the reader something they need to act on (a sibling to source alongside, the extraction harness to re-run, a join key) — not as a name-drop ("pairs with X", "feeds Y"). When in doubt, cut it. Every retained path must earn its place; collect the genuinely useful ones in a section-final References block rather than sprinkling them through prose.
• Prose style: follow knowledge-base/topics/writing-style.md for every deliverable (README and review.md). In short — lead each section with plain-language prose before any list; bullets only for genuine lists of parallel items, reasoning goes in prose; impersonal voice; plain language first with load-bearing terms glossed and linked to the glossary; keep file paths and identifiers out of prose and collect them in a References block at the end of the section; mermaid diagrams only when they cut reading load.