monitor-restore-unified

name

monitor-restore-unified

description

Re-register the every-3-hours UNIFIED OPS TICK cron — the CURRENT operator-owned monitor for the qwen3.5-122b-131k-datagen-opencode campaign (keep-3 datagen with autonomous rescue+refill) AND the delphi midtrain 1e23_p33m67_k0p20 (monitor-only), plus a marin+CoreWeave sweep and standing Daytona snapshot cleanup. The cron is session-only and recurring crons auto-expire after 7 days, so it's routinely lost on a session restart. Use at the start of a new session, after a disconnect/restart, or when the user asks to restore the monitor cron. Supersedes monitor-restore-iris (which is the OLDER datagen-OUT-of-scope variant); use THIS skill when the current session owns the 131k datagen campaign + midtrain. The per-tick sweep methodology lives in monitor-cron-sweep-iris; the table format in monitor-job-tables.

monitor-restore-unified

📍 Iris orientation — read first. Before acting, read the Iris tools catalog (.claude/ops/iris/iris_tools.md) and the Iris ops directory (.claude/ops/iris/ — iris_job_lifecycle.md for the TPU marin particulars, coreweave_gpu_ops.md for GPU). They carry the binding access/preamble/gotchas and the helper-script inventory below.

The recurring UNIFIED OPS TICK cron is session-only (lives in the Claude session, not on disk) and recurring crons auto-expire after 7 days — so it is routinely lost on a session restart. This skill is the durable source of truth for re-creating it: the canonical cron prompt below is what gets (re-)installed — copy it verbatim into CronCreate. The per-tick sweep methodology is monitor-cron-sweep-iris; the table format is monitor-job-tables.

Scope — how this differs from monitor-restore-iris

monitor-restore-iris installs the OLDER prompt where datagen is OUT of scope (a different agent owned it; that monitor covered eval + Levanter training + GPU-RL only). This skill is the opposite: the current session OWNS

• (B) the qwen3.5-122b-131k-datagen-opencode-iris campaign — keep-3 steady-state, with autonomous TERMINAL rescue (→ HF with literals) + refill, and confirmed-wedged kill+refill.
• (C) the delphi midtrain 1e23_p33m67_k0p20 — monitor-only.
• (A) the marin+CoreWeave sweep + native-route check + standing Daytona cleanup.

Install whichever matches who owns datagen this session. If both are somehow live, they conflict on datagen — keep only one (delete the other via CronDelete).

When to run

• Start of a new session where the 131k datagen campaign / midtrain are in flight.
• The user says the monitor/cron is gone, down, "not firing," or after a disconnect.
• After ~7 days (expiry).

Steps

1. Check if it already exists — call CronList. If a recurring job whose prompt begins "UNIFIED OPS TICK" is present, do nothing (a duplicate causes redundant SQL/tunnel load). If a stale variant (e.g. the datagen-out-of-scope monitor-restore-iris prompt) is present and this session owns datagen, CronDelete it and install the prompt below.
2. If absent, call CronCreate with:
   • cron: 23 */3 * * * (every 3 h at :23 — off the :00/:30 marks)
   • recurring: true
   • prompt: the exact text in the fenced block below.
3. Tell the user the new job id + the two caveats: session-only (dies when this Claude session exits — re-run this skill next session) and 7-day auto-expiry.

Notes

• durable: true is NOT honored in this harness (still session-only) — this skill IS the persistence layer.
• The cron only fires while the REPL is idle (not mid-task).
• Liveness tooling (operator directive 2026-07-13, memory [[iris_liveness_tooling_not_logtail]]): judge liveness/wedge via scripts/iris/watch_job_state.py + direct iris SQL, and progress via GCS artifacts (checkpoint step+ts for training, trial count / output-bucket population for datagen). NEVER diagnose liveness from raw iris job logs tail (interleaved multi-rank lines; --no-tail returns startup lines; a clean preempt emits no terminal log line).
• cgroup memory tracking is RETIRED (2026-07-13): midtrain (C) reports child state + checkpoint-step delta only — no memory-bounded field.
• Single-region output migration (OT-Agent c76dd23a): new datagen launches pass NO --gcs-output-dir, so the launcher's region-pin routes output to a co-located single-region bucket (gs://marin-us-<region>/ot-agent/<job>). Rescue resolves each job's RECORDED output URI via hpc.iris.job_output_resolver — legacy jobs stay on multi-region marin-models-{us,eu}. NEVER hardcode an output bucket.
• Companion skills: monitor-cron-sweep-iris (the tick methodology), monitor-job-tables (the datagen box table WITH Mean column), datagen-job-cleanup (canonical idempotent rescue for a TERMINAL/wedged arm — dispatch a subagent armed with it), datagen-launch-iris (refill).

Canonical cron prompt (copy verbatim into CronCreate)

UNIFIED OPS TICK. Source /Users/benjaminfeuer/Documents/secrets.env (never echo secrets). Spawn a general-purpose agent that reads the absolute-path ops docs and reports back; use PATH /Users/benjaminfeuer/miniconda3/envs/otagent/bin for iris+python. Wrap cw kubectl and any recursive gsutil in timeout. Use ABSOLUTE paths everywhere. Format the campaign status as a DATAGEN box-drawing table WITH a Mean-reward column per /Users/benjaminfeuer/Documents/OpenThoughts-Agent/.claude/skills/monitor-job-tables.

LIVENESS TOOLING (operator directive 2026-07-13): judge job liveness/wedge via /Users/benjaminfeuer/Documents/OpenThoughts-Agent/scripts/iris/watch_job_state.py + direct iris SQL query, and progress via GCS artifacts (checkpoint step+ts for training, trial count / output-bucket population for datagen). NEVER diagnose liveness from raw `iris job logs` tail (interleaved multi-rank lines; --no-tail returns STARTUP lines not latest; a clean preempt emits no terminal log line). See memory iris_liveness_tooling_not_logtail.

(A) SWEEP both iris clusters (marin GCP + cw-us-east-02a) and the native route (expect 401). Note Daytona snapshot count on the cli org. STANDING CLEANUP: each tick, proactively reclaim idle harbor__ snapshots >120min via /Users/benjaminfeuer/Documents/OpenThoughts-Agent/scripts/daytona/daytona_snapshot_manager.py --api-key-env DAYTONA_API_KEY --stale-days 0.0833 --delete-stale --yes (run from the OT-Agent dir, secrets sourced). Deletes ONLY idle harbor__ env snapshots; the --name-prefix harbor__ default GUARDS base images (daytonaio/sandbox:*, daytona-*, windows-*) — NEVER delete those, never ACTIVE-recent (<120min). Report before/after count.

(B) CAMPAIGN qwen3.5-122b-131k-datagen-opencode-iris — keep-3 steady-state. Box table (arm | dataset | completed/total | Mean | liveness). Confirm each arm RUNNING+serving+advancing (advancing = GCS trial count up vs last tick), single serve dir, no job.py:263.
  >=95% KILL-AND-HARVEST (operator directive 2026-07-14): if a state-3 RUNNING arm is past 95% completed/total on ANY tick, KILL the child (autonomously authorized; child only) and HARVEST it as a terminal arm right now — do NOT await the last stragglers (the long tail is not worth the held v5p-8). Treat it exactly like a >=60% terminal arm: rescue via datagen-job-cleanup + refill. (Applies to normal deterministic-verifier arms; a still-cold-compiling or resume-scanning arm that only shows 95% because its total is not yet known is exempt — require real advancing trials.)
  TERMINAL arm (state 4/5/6): if >=60% complete, dispatch a subagent ARMED WITH /Users/benjaminfeuer/Documents/OpenThoughts-Agent/.claude/skills/datagen-job-cleanup (read that SKILL.md, follow its idempotent steps): gs:// rescue of the OUTER recorded output dir (resolve via hpc.iris.job_output_resolver — single-region gs://marin-<region>/ot-agent/<job> for new jobs, multi-region gs://marin-models-{us,eu}/ for legacy; NEVER hardcode) so logs/ literals ride along -> avg_turns realness check -> HF upload via make_and_upload_trace_dataset.py --episodes last with literals AUTO-INCLUDED and --served_model Qwen/Qwen3.5-122B-A10B-FP8 -> verify HF non-empty + Literal yield X/Y (X>0) + count_populated_literal_rows>0 -> report rows+yield. (If <60%, resume instead.) THEN submit a refill (next un-launched tracker dataset) on the newest validated :tpu image --preemptible (no --gcs-output-dir, so the single-region region-pin engages) and let iris schedule; update the tracker with the cleanup (repo+rows) and the refill.
  CONFIRMED-WEDGED datagen (state 3 RUNNING but authoritative-state RUNNING AND 0 trials / empty output bucket for hours with no engine-serving marker, OR harbor frozen >=3h fd-monitor-only per datagen-job-cleanup) -> kill the child + refill (autonomously authorized; child only). Distinguish from a healthy cold-compile/resume-scan (engine bringing up / recompiling) — do NOT kill those.
  If keep-N<3 for any reason, submit refill(s), let iris schedule — do NOT gate on a capacity guess. Preemptible jobs stay preemptible.

(C) MIDTRAIN 1e23_p33m67_k0p20 (you OWN this, monitor-only) — ONE-LINE STATUS via watch_job_state.py/SQL (NOT log-tail): child state + live step / newest TEMP checkpoint step+ts (gs://marin-us-east5/tmp/ttl=14d/checkpoints-temp/.../delphi-1e23-p33m67-k0p20-lr0.67-b6607e/checkpoints/step-*/metadata.json) + whether the checkpoint step ADVANCED since last tick. NO memory-bounded field (cgroup tracking RETIRED 2026-07-13). A PENDING/re-placing child (preempt/crash-teardown/tier-monotonicity/capacity) is NORMAL — state it, do NOT flag/escalate/bounce; a stale checkpoint under preempt churn is expected. ONLY act on: a confirmed state-3 RUNNING-but-frozen wedge (authoritative state RUNNING + checkpoint stalled across multiple ticks + pod mismatch) -> Option A bounce the WEDGED CHILD only (never coordinator); or a confirmed cgroup OOM.

Relay a tight A/B/C report (campaign as a box-drawing datagen table with Mean).

If you change the cadence or scope, update BOTH the cron/prompt above and the live job (delete + recreate), and keep monitor-cron-sweep-iris (the procedure) in sync — so this skill stays the canonical copy.