raincloud

Walk through adding a new dataset to sources.json and producing its first build. Use when the user wants to onboard a new upstream source, add a slug to the manifest, or extend the catalog.

Walk through writing a new transform handler under scripts/pipeline/handlers/. Use when the default tighten_types/identity paths can't produce the right shape — row-level JSON parsing, streaming to avoid OOM, multi-output splitting, or VARIANT-from-the-start.

Load a Raincloud dataset via the lightweight Python loader API (`raincloud.load(slug)`). Use when the user wants to read a prepared parquet/vortex artifact, inspect a slug's metadata, or pull a dataset for downstream analysis from cache → mirror → local build.

Push locally-built Raincloud artifacts to a mirror (`scripts.pipeline.publish`). Use when the maintainer wants to upload one or more slugs' parquet/vortex bytes to a configured mirror after a successful build, gated on the snapshot's recorded sha256.

Use when the user asks to compute or refresh per-column statistics for a raincloud dataset — produces `outputs/v1/<slug>/profile.json` for the TUI's detail pane and `list_datasets --inspect`.

Run the full Raincloud pipeline (fetch → extract → parse → transform → write → validate → convert) for one or more dataset slugs. Use when the user asks to build a dataset, rebuild a slug, or process a batch.

Run only stage 7 — emit a sibling .vortex next to each opted-in parquet. Use when the user asks to (re)convert parquet files to Vortex format without rebuilding from raw bytes.

Use when the user wants to find "interesting" datasets — exposes the new tag / showcase / size / trait / view filters on list_datasets.

Regenerate the derived docs (datasets.md, handlers.md, snapshot.json). Use after a build, a manifest edit, or a handler add/remove/rename. snapshot.json is load-bearing — it's the fallback both for the TUI's columns / types modals AND for `datasets.md` regen on slugs not built locally, so partial-build maintainers don't dash-out the table. Other catalog views (columns, coverage, vortex-skip, hydrate candidates) are queryable via `/raincloud-list-datasets` and the TUI.

Run a memory- or runtime-heavy build safely with memory caps, scratch redirection, nohup, and progress logging. Use for multi-hour or multi-GB builds (JSONBench 100M, Wikipedia Structured Contents, OSM Germany, Public BI batches).

Filter and list datasets from sources.json without grepping a 545 KB JSON file. Use when the user asks "which slugs use handler X", "show me all UCI datasets", "what's gated behind Kaggle ToS", or any other catalog-shape question that's faster than reading docs/v1/datasets.md end to end.

Report per-dataset state (raw / workdir / parquet / vortex / variant-pending) across the manifest. Use when the user asks what's downloaded, what's built, what's missing, what needs re-tightening, or to triage which slugs still need work.

Static checks for sources.json — JSON Schema shape + cross-checks (handler registry, slug uniqueness, fetch.type/auth consistency). Use after any manifest edit, before triggering a build, or whenever an agent wants a sub-second sanity check that the manifest is well-formed.

Add a Kaggle dataset gated behind a one-time ToS click-through. Use when a Kaggle URL returns 403 from the API and the user needs the manifest+workflow pattern that documents the click-through gate cleanly.

Diagnostic checklist for a failing build — isolate which stage broke. Use when a build errors out, when validation fails, when fetch returns 403, or any time the user needs to triage a pipeline failure.

Run only the extract stage (unpack archives into _workdir/) for the given slugs. Use when the user wants to inspect what the unpack stage produces, or debug a handler complaint about missing files.

Run only the fetch stage (download raw bytes) for the given slugs. Use when the user wants to prime the cache, debug fetch logic, or download upstream bytes without running the full pipeline.

Run the optional hydrate stage to dereference a slug's URL column into a sibling parquet under outputs/v{n}/<slug>/parquet-hydrated/. Use ONLY for slugs that opted in via the manifest's `hydrate` block (see /raincloud-list-datasets --hydrate). Strong safety filter is on by default; bypass requires two flags. Makes outbound HTTP requests to arbitrary URLs from the open web.

Pick the right path (in-place tightening vs new-build streaming handler) for promoting a JSON column to VARIANT. Use when the user wants to upgrade a JSON-annotated string column to VARIANT and isn't sure whether to rewrite an existing parquet or change the build path.

Remove a dataset from sources.json and clean up its outputs. Use when the user wants to delete a slug from the manifest. Destructive — always confirm before acting.

In-place promote JSON-annotated string columns to VARIANT in built parquets. Use when a parquet already exists and you want to upgrade its JSON columns without rerunning the full pipeline.