메뉴
Report per-dataset state (raw / workdir / parquet / vortex / variant-pending) across the manifest. Use when the user asks what's downloaded, what's built, what's missing, what needs re-tightening, or to triage which slugs still need work.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Walk through adding a new dataset to sources.json and producing its first build. Use when the user wants to onboard a new upstream source, add a slug to the manifest, or extend the catalog.
Walk through writing a new transform handler under scripts/pipeline/handlers/. Use when the default tighten_types/identity paths can't produce the right shape — row-level JSON parsing, streaming to avoid OOM, multi-output splitting, or VARIANT-from-the-start.
Load a Raincloud dataset via the lightweight Python loader API (`raincloud.load(slug)`). Use when the user wants to read a prepared parquet/vortex artifact, inspect a slug's metadata, or pull a dataset for downstream analysis from cache → mirror → local build.
Push locally-built Raincloud artifacts to a mirror (`scripts.pipeline.publish`). Use when the maintainer wants to upload one or more slugs' parquet/vortex bytes to a configured mirror after a successful build, gated on the snapshot's recorded sha256.
Use when the user asks to compute or refresh per-column statistics for a raincloud dataset — produces `outputs/v1/<slug>/profile.json` for the TUI's detail pane and `list_datasets --inspect`.
Run the full Raincloud pipeline (fetch → extract → parse → transform → write → validate → convert) for one or more dataset slugs. Use when the user asks to build a dataset, rebuild a slug, or process a batch.
| name | raincloud-status |
| description | Report per-dataset state (raw / workdir / parquet / vortex / variant-pending) across the manifest. Use when the user asks what's downloaded, what's built, what's missing, what needs re-tightening, or to triage which slugs still need work. |
Run the status reporter:
python -m scripts.pipeline.status $ARGUMENTS
Walks the manifest and reports per-slug filesystem state in five columns:
| Column | What it shows |
|---|---|
raw | outputs/raw_downloads/<slug>/ present. ≠ if fetch.expected_bytes is declared and the on-disk total disagrees (only checked for single-URL specs — that's the same condition fetch.py uses). |
work | _workdir/<slug>/ non-empty (extract scratch — wiped by --clean-workdir). |
parquet | outputs/v{n}/<slug>/<slug>.parquet present. Shows row count; ≠ prefix if it disagrees with expect.rows. |
vortex | Sibling .vortex for slugs with convert.vortex: true. n/a for non-opted-in specs, stale if the parquet is newer. |
variant | Count of residual JSON-annotated columns (= /raincloud-tighten-variant pending). ? when --fast skipped the parquet footer scan. |
Selection (default: every slug in the manifest):
<slug>... — positional slugs--all — explicit "every dataset" (the default, kept for parity with /raincloud-build)Modifiers:
--fast — skip the parquet footer scan. Drops the row count and JSON-column check; keeps everything else. Useful on slow storage or for a quick pass over the full catalog.--missing-only — narrow output to slugs with at least one incomplete stage. Combines well with --fast for a quick "what still needs work" view.--json — emit a JSON array instead of the table. Use when piping into another tool.The summary line at the bottom shows totals, e.g.:
N slugs · raw M/N · parquet M/N · rows-match M/N · vortex M/N · tighten-variant pending K
This is read-only — it never touches outputs/ or _workdir/. Safe to invoke whenever you want a snapshot. Suggest /raincloud-build for missing parquets, /raincloud-convert for missing vortex siblings, /raincloud-tighten-variant for variant-pending slugs.
Context: SKILLS.md, AGENTS.md "How the build pipeline is structured".