菜单
Run a memory- or runtime-heavy build safely with memory caps, scratch redirection, nohup, and progress logging. Use for multi-hour or multi-GB builds (JSONBench 100M, Wikipedia Structured Contents, OSM Germany, Public BI batches).
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
Walk through adding a new dataset to sources.json and producing its first build. Use when the user wants to onboard a new upstream source, add a slug to the manifest, or extend the catalog.
Walk through writing a new transform handler under scripts/pipeline/handlers/. Use when the default tighten_types/identity paths can't produce the right shape — row-level JSON parsing, streaming to avoid OOM, multi-output splitting, or VARIANT-from-the-start.
Load a Raincloud dataset via the lightweight Python loader API (`raincloud.load(slug)`). Use when the user wants to read a prepared parquet/vortex artifact, inspect a slug's metadata, or pull a dataset for downstream analysis from cache → mirror → local build.
Push locally-built Raincloud artifacts to a mirror (`scripts.pipeline.publish`). Use when the maintainer wants to upload one or more slugs' parquet/vortex bytes to a configured mirror after a successful build, gated on the snapshot's recorded sha256.
Use when the user asks to compute or refresh per-column statistics for a raincloud dataset — produces `outputs/v1/<slug>/profile.json` for the TUI's detail pane and `list_datasets --inspect`.
Run the full Raincloud pipeline (fetch → extract → parse → transform → write → validate → convert) for one or more dataset slugs. Use when the user asks to build a dataset, rebuild a slug, or process a batch.
基于 SOC 职业分类
| name | raincloud-large-build |
| description | Run a memory- or runtime-heavy build safely with memory caps, scratch redirection, nohup, and progress logging. Use for multi-hour or multi-GB builds (JSONBench 100M, Wikipedia Structured Contents, OSM Germany, Public BI batches). |
| argument-hint | <slug>... [--all] [--loose] [--clean-workdir] |
| disable-model-invocation | true |
Run a memory- or runtime-heavy build with the safety knobs enabled. Reference: SKILLS.md "Running a large build safely".
Confirm with the user before triggering. Observed timings: JSONBench 100M ≈ 6 h, Wikipedia Structured Contents → 34 GB parquet (multi-hour), OSM Germany ≈ 45 min per kind. Rebuilding wipes and redoes the existing outputs/v1/<slug>/<slug>.parquet+.vortex pair on disk.
Pattern (adjust the slug, memory cap, and tempdir to the user's machine):
RAINCLOUD_DUCKDB_MEMORY_LIMIT=32GB \
RAINCLOUD_DUCKDB_TEMP_DIRECTORY=/mnt/scratch/duckdb-tmp \
PYTHONUNBUFFERED=1 \
nohup python -m scripts.pipeline.build $ARGUMENTS \
> /tmp/raincloud-build.log 2>&1 &
Flag rationale:
--loose — first build of a new slug, before expect.rows is known. Downgrades row-count mismatches from errors to warnings.--clean-workdir — wipe _workdir/<slug>/ after each successful build. Essential for large batch runs (Public BI decompressed CSVs can hit ~100 GB).RAINCLOUD_DUCKDB_MEMORY_LIMIT — caps DuckDB's working set; default (~80% of system RAM) can swap-thrash on heavily-nested VARIANT shredding. 96 GB is the tested ceiling for Open Food Facts.RAINCLOUD_DUCKDB_TEMP_DIRECTORY — point at a large volume; the system tempdir often runs out on big builds.PYTHONUNBUFFERED=1 — log file flushes line-by-line so progress is inspectable mid-run.nohup … & — survives terminal disconnect.Monitor:
tail -f /tmp/raincloud-build.log
du -sh _workdir/<slug>/ # scratch growth
df -h . # disk headroom
Do not invoke this without confirming the slug and timing with the user. Suggest /raincloud-docs after success.