القائمة
Walk through adding a new dataset to sources.json and producing its first build. Use when the user wants to onboard a new upstream source, add a slug to the manifest, or extend the catalog.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
Walk through writing a new transform handler under scripts/pipeline/handlers/. Use when the default tighten_types/identity paths can't produce the right shape — row-level JSON parsing, streaming to avoid OOM, multi-output splitting, or VARIANT-from-the-start.
Load a Raincloud dataset via the lightweight Python loader API (`raincloud.load(slug)`). Use when the user wants to read a prepared parquet/vortex artifact, inspect a slug's metadata, or pull a dataset for downstream analysis from cache → mirror → local build.
Push locally-built Raincloud artifacts to a mirror (`scripts.pipeline.publish`). Use when the maintainer wants to upload one or more slugs' parquet/vortex bytes to a configured mirror after a successful build, gated on the snapshot's recorded sha256.
Use when the user asks to compute or refresh per-column statistics for a raincloud dataset — produces `outputs/v1/<slug>/profile.json` for the TUI's detail pane and `list_datasets --inspect`.
Run the full Raincloud pipeline (fetch → extract → parse → transform → write → validate → convert) for one or more dataset slugs. Use when the user asks to build a dataset, rebuild a slug, or process a batch.
Run only stage 7 — emit a sibling .vortex next to each opted-in parquet. Use when the user asks to (re)convert parquet files to Vortex format without rebuilding from raw bytes.
| name | raincloud-add-dataset |
| description | Walk through adding a new dataset to sources.json and producing its first build. Use when the user wants to onboard a new upstream source, add a slug to the manifest, or extend the catalog. |
Guide the user through adding a new dataset entry. Reference: SKILLS.md "Adding a new dataset", sources.schema.md.
Steps:
Identify the upstream. Confirm with the user:
source_url.expect.rows; can be null on first build).Append a DatasetSpec to sources.json using the Python load-edit-dump pattern from AGENTS.md — never sed. Start from templates/minimal_spec.json (every field present with placeholder values) rather than typing one from scratch. Minimal direct-HTTP shape:
{
"slug": "my-dataset",
"short_name": "My Dataset",
"full_name": "My Dataset (publisher attribution)",
"description": "One-line summary.",
"license": { "spdx": "CC0-1.0", "source_url": "...", "redistribution_permitted": true, "attribution_required": false },
"fetch": { "type": "http", "urls": ["https://..."], "auth": null },
"extract": { "type": "passthrough" },
"parse": { "reader": "csv", "options": { "delimiter": "," } },
"transform": { "handler": "tighten_types", "params": {} },
"write": { "output": "my-dataset.parquet", "compression": "zstd", "row_group_size_rows": 1048576, "statistics": true },
"expect": { "rows": 123456 }
}
Validate the manifest. Invoke /raincloud-validate-manifest — sub-second check that the new entry has the right shape, the handler resolves, the slug is unique, and fetch.type/fetch.auth agree. Catches typos before paying for a fetch.
Run the first build. Invoke /raincloud-build <slug> --loose (the --loose is essential when the row count is a guess). If expect.rows was wrong, update the manifest with the actual count once the build succeeds.
Regenerate docs. Invoke /raincloud-docs (or just /raincloud-docs datasets columns_parquet coverage_parquet).
Optionally opt into Vortex. If the dataset's types are vortex-compatible, add "convert": { "vortex": true } to the spec and run /raincloud-convert <slug>.
If the upstream needs unpacking, set extract.type accordingly and pick a parser — see existing specs in sources.json for shapes. If the source has nested JSON or row-level processing, you'll likely need a custom handler — see /raincloud-add-handler. If it's a Kaggle dataset behind ToS acceptance, see /raincloud-add-kaggle-tos.