Menu
Use when working on bootstrap-lab, warehouse management, seed export or restore, legacy raw/parquet cleanup, or canonical data layout in polymarket-tmax-lab.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Use when running the LGBM autoresearch loop, editing candidate YAML specs, gating new variants, or promoting winners in polymarket-tmax-lab.
Use when working on bootstrap-lab, warehouse management, seed export or restore, legacy raw/parquet cleanup, or canonical data layout in polymarket-tmax-lab.
Use when working on dataset materialization, model training, backtesting, scan-edge signals, paper-trading workflows, or experiment artifacts in polymarket-tmax-lab.
Use when working on dataset materialization, model training, backtesting, scan-edge signals, paper-trading workflows, or experiment artifacts in polymarket-tmax-lab.
Use when running the LGBM autoresearch loop, editing candidate YAML specs, gating new variants, or promoting winners in polymarket-tmax-lab.
Use when changing Polymarket discovery, temperature market filtering, rule parsing, settlement source handling, station mapping, or official truth adapters.
Based on SOC occupation classification
| name | pmtmax-data-ops |
| description | Use when working on bootstrap-lab, warehouse management, seed export or restore, legacy raw/parquet cleanup, or canonical data layout in polymarket-tmax-lab. |
Use this skill for storage, bootstrap, and long-history data operations.
AGENTS.md.docs/codebase/storage-and-configs.md.docs/agent-skills/data-ops.mddocs/agent-skills/safety-and-rules.mdweather_train weather-real station/date pretrain databootstrap-lab, export-seed, restore-seedinventory-legacy-runs, archive-legacy-runsconfigs/market_inventory/historical_temperature_snapshots.jsonconfigs/market_inventory/full_training_set_snapshots.jsonhistorical_training_set* / historical_backtest_panel overwrite is opt-in only.synthetic_ market ids, fixture forecasts, and fabricated books are trust violations.weather_train uses PMTMAX_DATASET_PROFILE=weather_real and must not contain Polymarket market ids, rule JSON, CLOB books, price history, or publish evidence.historical_real / benchmark / publish commands remain PMTMAX_DATASET_PROFILE=real_market.trust-check before canonical overwrite, long collection, benchmark, or release validation.--http-timeout-seconds, --http-retries,
--http-retry-wait-min-seconds, --http-retry-wait-max-seconds, plus
stderr progress by default. Keep --max-consecutive-429 2: two consecutive
Open-Meteo 429 responses mean the free-path daily limit is effectively hit.backfill-forecasts also supports --max-consecutive-429 2.
If two consecutive Open-Meteo 429 responses appear, the run flushes already
collected rows, cancels, and should be recorded in checker state as
rate-limit-cancelled; do not immediately retry the same free-path job.scripts/pmtmax-workspace weather_train uv run python scripts/run_weather_train_queue_agent.py
It reads checker/weather_train_status.md, advances the next 7-day chunk,
updates checker markdown after every chunk, and auto-refreshes
gaussian_emos pretrain when the configured row-gap threshold is met. When
it sees two consecutive 429 responses, it cancels the remaining chunk,
records rate-limit-cancelled, and waits for cooldown/API-key collection.scripts/pmtmax-workspace historical_real uv run python scripts/run_historical_price_recovery_agent.py
It reads checker/historical_price_status.md, advances the next shard,
rebuilds the canonical backtest panel, refreshes the latest coverage JSON,
and updates the historical-price checker markdown files.weather_train gold has 11,499 rows. Full successful ranges are
2024-01-03..2024-05-24, 2024-06-01..2024-12-30, and
2026-01-01..2026-01-14; partial coverage includes
2024-12-31..2025-01-06 and 2026-01-15..2026-01-21.
2025-01-07..2025-01-27 and 2026-01-22..2026-01-28 are retry-only/free-path
daily-limit ranges. Read the live row count from checker/weather_train_status.md.historical_real collection state on April 27, 2026:
discovery120 appended 1 Tokyo event URL after the completed discovery80
batch, merged the local curated backlog to 2,602 snapshots, and then
promoted that verified backlog into full_training_set_snapshots.json as a
sharded manifest plus full_training_set_snapshots.d/part-000..003.json.
Canonical v2 historical_training_set now has 7,794 rows / 2,598
markets, and canonical v2 historical_backtest_panel has 68,550 token rows
with 30,615 ok. No two-consecutive-429 cancellation occurred, and
follow-up discovery160 appended 0 URLs; a later broader discovery appended
22 URL-manifest entries that remain next-round snapshot backlog.
Read live details from checker/historical_real_status.md.checker/weather_train_status.md and
checker/weather_train_collection_log.md synchronized after every
collection/pretrain turn.checker/historical_price_status.md and
checker/historical_price_collection_log.md synchronized after every daily
price recovery turn.weather_train queue collection can run at the same time
as historical_real price-history collection because they write to separate
workspace roots and hit different upstream APIs. Do not run two mutating
historical_real jobs at once; backfill-price-history,
materialize-backtest-panel, build-dataset, and backfill-forecasts
should stay serialized within that workspace.--output-name values by default; only pass --allow-canonical-overwrite for intentional canonical promotion.artifacts/recovery/ first.--truth-no-cache is the real pmtmax flag; use it when cached truth payloads look stale or malformed. --truth-per-source-limit 1 is NOT a pmtmax CLI flag — it is exposed only by the historical refresh/inventory helper scripts (scripts/run_historical_refresh_pipeline.sh, scripts/build_historical_market_inventory.py, scripts/validate_historical_market_inventory.py, scripts/refresh_historical_event_urls.py).historical_temperature_snapshots.json is the canonical curated backlog for data collection. Use this when the task is “collect the latest historical markets”.full_training_set_snapshots.json is a checked-in training inventory, not an auto-refreshed mirror of the canonical historical backlog.full_training_set_snapshots.json is now a sharded manifest for the verified 2,602-snapshot curated backlog. Canonical v2 materializes 7,794 training rows / 2,598 markets; the legacy gold path may still show the old 5,478 row / 1,826 market dataset.ops_daily collection records forward evidence only; it does not auto-append to full_training_set_snapshots.json or retrain the champion.backfill-forecasts --missing-only so the run only fetches forecast request keys that are absent from bronze_forecast_requests.--model explicitly on
backfill-forecasts, build-dataset, or materialize-training-set.
The base config can default to gfs_seamless; a curated rebuild without
explicit models can keep the same row count but silently become GFS-only.0C / 32F forecasts. After changing
this logic, rebuild a non-canonical variant and compare sentinel/source
availability counts before training.scripts/pmtmax-workspace weather_train uv run pmtmax collect-weather-training --date-from 2026-01-22 --date-to 2026-01-22 --model gfs_seamless --missing-only --rate-limit-profile free --http-timeout-seconds 15 --http-retries 1 --http-retry-wait-min-seconds 1 --http-retry-wait-max-seconds 8scripts/pmtmax-workspace historical_real uv run pmtmax backfill-forecasts --markets-path configs/market_inventory/historical_temperature_snapshots.json --strict-archive --missing-onlyscripts/pmtmax-workspace historical_real uv run pmtmax backfill-forecasts --markets-path configs/market_inventory/historical_temperature_snapshots.json --model ecmwf_ifs025 --strict-archive --missing-only --single-run-horizon market_open --single-run-horizon previous_evening --single-run-horizon morning_of --max-consecutive-429 2scripts/pmtmax-workspace historical_real uv run pmtmax materialize-training-set --markets-path configs/market_inventory/historical_temperature_snapshots.json --model ecmwf_ifs025 --model ecmwf_aifs025_single --model kma_gdps --model gfs_seamless --output-name <variant>scripts/pmtmax-workspace historical_real uv run pmtmax build-dataset --markets-path configs/market_inventory/full_training_set_snapshots.json --forecast-missing-only --allow-canonical-overwritescripts/pmtmax-workspace historical_real uv run pmtmax backfill-price-history --markets-path configs/market_inventory/full_training_set_snapshots.json --only-missing --price-no-cache --limit-markets 25
scripts/pmtmax-workspace historical_real uv run pmtmax materialize-backtest-panel --markets-path configs/market_inventory/full_training_set_snapshots.json --allow-canonical-overwritescripts/pmtmax-workspace historical_real uv run pmtmax summarize-price-history-coverage --markets-path configs/market_inventory/full_training_set_snapshots.json
scripts/pmtmax-workspace historical_real uv run pmtmax backfill-price-history --markets-path configs/market_inventory/full_training_set_snapshots.json --only-missing --price-no-cache --limit-markets 25 --offset-markets 0
scripts/pmtmax-workspace historical_real uv run pmtmax materialize-backtest-panel --markets-path configs/market_inventory/full_training_set_snapshots.json --allow-canonical-overwrite
scripts/pmtmax-workspace historical_real uv run python scripts/run_historical_price_recovery_agent.pyuv run pmtmax bootstrap-lab --forecast-missing-onlyfull_training_set_snapshots.json does not regenerate itself from historical_temperature_snapshots.json.historical_temperature_snapshots.json, then explicitly curate and replace full_training_set_snapshots.json in the same change.restore-seed rebuilds the local warehouse from canonical parquet mirrors directly in DuckDB; it should not need pandas full-table reads.