메뉴
Walk through adding a new dataset to sources.json and producing its first build. Use when the user wants to onboard a new upstream source, add a slug to the manifest, or extend the catalog.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Walk through writing a new transform handler under scripts/pipeline/handlers/. Use when the default tighten_types/identity paths can't produce the right shape — row-level JSON parsing, streaming to avoid OOM, multi-output splitting, or VARIANT-from-the-start.
Load a Raincloud dataset via the lightweight Python loader API (`raincloud.load(slug)`). Use when the user wants to read a prepared parquet/vortex artifact, inspect a slug's metadata, or pull a dataset for downstream analysis from cache → mirror → local build.
Push locally-built Raincloud artifacts to a mirror (`scripts.pipeline.publish`). Use when the maintainer wants to upload one or more slugs' parquet/vortex bytes to a configured mirror after a successful build, gated on the snapshot's recorded sha256.
Use when the user asks to compute or refresh per-column statistics for a raincloud dataset — produces `outputs/v1/<slug>/profile.json` for the TUI's detail pane and `list_datasets --inspect`.
Run the full Raincloud pipeline (fetch → extract → parse → transform → write → validate → convert) for one or more dataset slugs. Use when the user asks to build a dataset, rebuild a slug, or process a batch.
Run only stage 7 — emit a sibling .vortex next to each opted-in parquet. Use when the user asks to (re)convert parquet files to Vortex format without rebuilding from raw bytes.
| name | raincloud-add-dataset |
| description | Walk through adding a new dataset to sources.json and producing its first build. Use when the user wants to onboard a new upstream source, add a slug to the manifest, or extend the catalog. |
Guide the user through adding a new dataset entry. Reference: SKILLS.md "Adding a new dataset", sources.schema.md.
Steps:
Identify the upstream. Confirm with the user:
source_url.expect.rows; can be null on first build).Append a DatasetSpec to sources.json using the Python load-edit-dump pattern from AGENTS.md — never sed. Start from templates/minimal_spec.json (every field present with placeholder values) rather than typing one from scratch. Minimal direct-HTTP shape:
{
"slug": "my-dataset",
"short_name": "My Dataset",
"full_name": "My Dataset (publisher attribution)",
"description": "One-line summary.",
"license": { "spdx": "CC0-1.0", "source_url": "...", "redistribution_permitted": true, "attribution_required": false },
"fetch": { "type": "http", "urls": ["https://..."], "auth": null },
"extract": { "type": "passthrough" },
"parse": { "reader": "csv", "options": { "delimiter": "," } },
"transform": { "handler": "tighten_types", "params": {} },
"write": { "output": "my-dataset.parquet", "compression": "zstd", "row_group_size_rows": 1048576, "statistics": true },
"expect": { "rows": 123456 }
}
Validate the manifest. Invoke /raincloud-validate-manifest — sub-second check that the new entry has the right shape, the handler resolves, the slug is unique, and fetch.type/fetch.auth agree. Catches typos before paying for a fetch.
Run the first build. Invoke /raincloud-build <slug> --loose (the --loose is essential when the row count is a guess). If expect.rows was wrong, update the manifest with the actual count once the build succeeds.
Regenerate docs. Invoke /raincloud-docs (or just /raincloud-docs datasets columns_parquet coverage_parquet).
Optionally opt into Vortex. If the dataset's types are vortex-compatible, add "convert": { "vortex": true } to the spec and run /raincloud-convert <slug>.
If the upstream needs unpacking, set extract.type accordingly and pick a parser — see existing specs in sources.json for shapes. If the source has nested JSON or row-level processing, you'll likely need a custom handler — see /raincloud-add-handler. If it's a Kaggle dataset behind ToS acceptance, see /raincloud-add-kaggle-tos.