| name | dataset-compile |
| description | Manufacture a structured staging dataset by compiling many public sources (award lists, registries, reports, portals) into schema'd, provenance-flagged, verified records — for cases where no single external dataset exists to adopt. Use whenever the user wants to compile, harvest, scrape, assemble, or "go collect examples/records" from scattered public sources; build a dataset that doesn't exist yet; or turn many announcements into one queryable table — even if they never say "compile". Sibling to dataset-discovery: discovery *finds* an existing dataset, compile *builds* one. Writes only to dataset-compile/<slug>/, never the knowledge base or the catalog.
|
Dataset Compile (manufacture)
Some datasets you need don't exist as a single publisher's file — they're
scattered across dozens of public pages (award announcements, priority lists,
recipient tables, PDFs). This skill compiles that scatter into one structured,
provenance-tracked, verified staging dataset, ready to promote into
dataset-review.
It is the sibling entry point to dataset-discovery in the pipeline
(discover | compile) -> review -> pipeline -> test. Discovery finds and
triages a dataset that already exists externally; compile assembles one that
doesn't, from many sources. Both hand off to review. This skill writes only
to dataset-compile/, never to knowledge-base/ (that's reference context) or
dataset-review/catalog/ (that's for vetted external datasets).
The worked reference instance is dataset-compile/mn-climate-awards/ (Minnesota
climate-finance awards, indexed by funding "route"). The method below is generic;
that folder is one set of values.
What makes this different from an ingested dataset
An ingested dataset has one publisher, one license, and a feed to poll — it
belongs in dataset-discovery -> dataset-review. A compiled dataset is
authored: no single source, synthesised across many, and it changes as your
research proceeds, not as an upstream feed updates. That's why it gets its own
lane and its own provenance discipline (below), and why it's a staging
dataset — not knowledge, not yet a vetted dataset.
Output contract
One folder per dataset: dataset-compile/<slug>/ containing:
schema.json — the contract: grain, primary key, scope, and every field
tagged sourced / derived / derived-by-rule / meta (see provenance).data.csv — one row per record, flat and diff-friendly. Schema lives in
schema.json, not here.collection.md — provenance & method: how it was compiled, the verification
status, an insights section, and the to-proper gap checklist.sources.yaml — the source registry. Since there's no feed, this list IS
the refresh mechanism; each entry carries a last_fetched date.raw/ — source files that had to be downloaded (e.g. PDFs). Kept for
re-parsing; URL-fetch-via-code may be unavailable, so a local copy matters.references/ — optional deterministic lookups the derived-by-rule fields use
(e.g. a place -> region map), kept auditable and separate from data.csv.
Skeletons for each are in references/templates.md.
The core design: rollup index + facet axes
Pick one rollup dimension that is the natural parent of the records and that
determines their shape — in the reference instance it's the funding route,
because the route dictates what a fundable application must contain. Then keep a
small set of facet fields so the data backs a general get_<records>(params)
query and generalises to a second domain as new rows, not a rebuild. The rollup
is a derived parent tag layered on top of the facets, never a replacement for
them. Expect the rollup to be lopsided — some values richly enumerable,
others thin. That lopsidedness is a finding, not a gap: it tells you where the
reachable, documented records actually are.
Workflow
1. Frame
Decide the grain (what is one row?), the rollup dimension, the facet fields, and
the query the dataset should answer. Write these into schema.json first — the
schema is the contract, and framing it up front stops the compile from drifting.
2. Navigate by an existing reference, don't compile blind
Check knowledge-base/ for a map of the domain that says where the records
live — the reference instance was steered entirely by a landscape doc that
listed each route's award-list sources. If no such map exists, note it, rely on
general search, and consider seeding one. Reference context navigates; it is not
the data.
3. Compile
Search per source category -> fetch the primary source -> extract records.
Prefer primary tables/PDFs over summaries. For PDFs, download to raw/ and parse
locally (pdftotext -layout, then regex/column logic); a bad fetch is often a
fetch artifact, not a bad file. Expect dead ends and thin routes — record them,
don't invent rows to fill them.
4. Structure + tag provenance
Map each record into the schema. For every field, decide its provenance and tag
it (next section). This is the discipline that keeps judgment from masquerading
as fact.
5. Verify — reconcile to control totals
The strongest QA available: if a source publishes an aggregate (a totals row, a
published count, subtotals per group), parse the rows and check your sums tie
to it. In the reference instance the 18 grant line-items reconciled to two
published subtotals, and 135 loan rows tied exactly (per group) to a totals-by-
part table — that's what makes those figures trustworthy. Where no control total
exists, cross-check individual rows against the source and mark the rest
verified: false. Never present an unverified figure as checked.
6. Insights pass
Compute what the data says (distributions, magnitudes, funnels, concentration)
and write it into collection.md. The compiled records usually reveal structure
the prose reference couldn't — capture it.
7. Record provenance
Fill collection.md (method, verification status, gaps) and sources.yaml
(every source + last_fetched). These are what let someone else re-run or trust
the compile.
Provenance discipline (the load-bearing rule)
Every field in schema.json is tagged:
sourced — read directly from a primary source (verifiable against a URL).derived — assigned by judgment, not stated in the source. Lower-trust; the
main work remaining before promotion is turning these into rules.derived-by-rule — computed by an explicit, auditable lookup or rule (e.g. a
references/*.csv map), not per-row judgment. Reproducible and correctable in
one place. Promote derived fields to this whenever you can.meta — bookkeeping (ids, verified flag, free-text notes).
Keep schema.json (contract) separate from data.csv (values), keep
derived-by-rule inputs in references/ so they're auditable, and treat
sources.yaml as the refresh mechanism.
Where it sits & promotion
Compile is pre-review staging. It graduates into dataset-review when: scope is
locked, derived fields are computed by rule, gaps are filled from primary
sources, and provenance is complete. Review then creates the catalog entry and,
downstream, the pipeline into production. Don't put a compiled file in the
knowledge base (it's not prose reference) or the catalog (it's not a vetted
external dataset) — the compile lane is exactly the missing middle.
Improving this skill
After a run, route lessons by the "how vs what" test:
- A fact about a specific source (a portal quirk, a PDF's column layout, where
a figure really lives) -> the dataset's own
collection.md / sources.yaml.
- General "where records of type X live" knowledge that any future compile
would reuse ->
knowledge-base/ as a concise reference map, added
deliberately, never auto-appended per run.
The method (this file + references/) stays generic. Most lessons aren't
reusable — don't pad it.
