raincloud-tighten-variant

name	raincloud-tighten-variant
description	In-place promote JSON-annotated string columns to VARIANT in built parquets. Use when a parquet already exists and you want to upgrade its JSON columns without rerunning the full pipeline.

Run the JSON → VARIANT in-place tightening pass:

python -m scripts.pipeline.tighten_variant $ARGUMENTS

Selection:

• (no slugs) — every parquet under outputs/v*/ is considered.
• <slug>... — only those specific slugs.

Modifiers:

• --dry-run — report which parquets would be rewritten, without touching them.
• --chunked — force row-group-at-a-time mode. Needed for wide / deeply-nested records where single-shot VARIANT cast spills blow up DuckDB memory (e.g. Open Food Facts).

Behavior:

• Idempotent: parquets already lacking JSON-annotated columns are skipped (reported as "already").
• Uses CAST(col AS VARIANT) + atomic tmp-rename via duckdb_connect (which applies storage_compatibility_version=v1.5.0).
• Memory-hungry on heavily-nested payloads. Set RAINCLOUD_DUCKDB_MEMORY_LIMIT=96GB for known-heavy cases (Open Food Facts is the tested ceiling).

When to use:

• After a build that produced a JSON-annotated string column you want to promote without rerunning the full pipeline.
• For new builds that should emit VARIANT from the start, write a streaming handler instead — see /raincloud-add-handler and SKILLS.md.

After this runs, suggest /raincloud-docs columns_parquet coverage_parquet to refresh the parquet-side derived docs.