monitor-restore-unified

name

monitor-restore-unified

description

Re-register the every-3-hours UNIFIED OPS TICK cron — the CURRENT operator-owned monitor for the qwen3.5-122b-131k-datagen-opencode campaign (keep-3 datagen with autonomous rescue+refill) AND the delphi midtrain 1e23_p33m67_k0p20 (monitor-only), plus a marin+CoreWeave sweep and standing Daytona snapshot cleanup. The cron is session-only and recurring crons auto-expire after 7 days, so it's routinely lost on a session restart. Use at the start of a new session, after a disconnect/restart, or when the user asks to restore the monitor cron. Supersedes monitor-restore-iris (which is the OLDER datagen-OUT-of-scope variant); use THIS skill when the current session owns the 131k datagen campaign + midtrain. The per-tick sweep methodology lives in monitor-cron-sweep-iris; the table format in monitor-job-tables.

monitor-restore-unified

Read first: Iris tools catalog (.agents/ops/iris/ops.md) and ops directory (.agents/ops/iris/ — ops.md for TPU marin particulars, ops.md for GPU) carry the binding access/preamble/gotchas + helper-script inventory.

The UNIFIED OPS TICK cron is session-only, expires after seven days, and may be lost on restart. Copy the canonical prompt below verbatim into CronCreate. For the per-tick method use monitor-cron-sweep-iris; for tables use monitor-job-tables.

Scope — supersedes monitor-restore-iris

monitor-restore-iris installs the older prompt where datagen is out of scope. This skill owns:

• (B) the qwen3.5-122b-131k-datagen-opencode-iris campaign — keep-3 steady-state, with autonomous TERMINAL rescue (→ HF with literals) + refill, and confirmed-wedged kill+refill.
• (C) the delphi midtrain 1e23_p33m67_k0p20 — monitor-only.
• (A) the marin+CoreWeave sweep + native-route check + standing Daytona cleanup.

If both prompts are live, keep only one (CronDelete the other).

When to run

• Start of a new session where the 131k datagen campaign / midtrain are in flight.
• The user says the monitor/cron is gone, down, "not firing," or after a disconnect/restart.
• After ~7 days (expiry).

Steps

1. Check if it exists — CronList. If a recurring prompt begins "UNIFIED OPS TICK", do nothing. If this session owns datagen and a stale monitor-restore-iris variant is present, CronDelete it and install the prompt below.
2. If absent, CronCreate with:
   • cron: 23 */3 * * * (every 3 h at :23 — off the :00/:30 marks)
   • recurring: true
   • prompt: the exact text in the fenced block below.
3. Tell the user the new job id, that it is session-only and expires after seven days. durable: true is not honored; the cron fires only while the REPL is idle.

Notes

• Liveness tooling (operator directive 2026-07-13, memory [[iris_liveness_tooling_not_logtail]]): judge liveness/wedge via scripts/iris/iris_ops.py + direct iris SQL, progress via GCS artifacts (checkpoint step+ts for training, trial count / output-bucket population for datagen). NEVER diagnose liveness from raw iris job logs tail (interleaved multi-rank lines; --no-tail returns startup lines; a clean preempt emits no terminal log line).
• cgroup memory tracking is RETIRED (2026-07-13): midtrain (C) reports child state + checkpoint-step delta only.
• Single-region output migration (OT-Agent c76dd23a): new datagen launches pass NO --gcs-output-dir, so the launcher's region-pin routes output to a co-located single-region bucket (gs://marin-us-<region>/ot-agent/<job>). Rescue resolves each job's RECORDED output URI via hpc.iris.job_output_resolver — legacy jobs stay on multi-region marin-models-{us,eu}. NEVER hardcode an output bucket.
• Companion skills: monitor-cron-sweep-iris (tick methodology), monitor-job-tables (datagen box table WITH Mean column), datagen-job-cleanup (idempotent rescue for a TERMINAL/wedged arm — dispatch a subagent armed with it), datagen-launch-iris (refill).

Canonical cron prompt (copy verbatim into CronCreate)

UNIFIED OPS TICK. Source /Users/benjaminfeuer/Documents/secrets.env (never echo secrets). Spawn a general-purpose agent that reads the absolute-path ops docs and reports back; use PATH /Users/benjaminfeuer/miniconda3/envs/otagent/bin for iris+python. Wrap cw kubectl and any recursive gsutil in timeout. Use ABSOLUTE paths everywhere. Format the campaign status as a DATAGEN box-drawing table WITH a Mean-reward column per /Users/benjaminfeuer/Documents/OpenThoughts-Agent/.agents/skills/monitor-job-tables.

LIVENESS TOOLING (operator directive 2026-07-13): judge job liveness/wedge via /Users/benjaminfeuer/Documents/OpenThoughts-Agent/scripts/iris/iris_ops.py + direct iris SQL query, and progress via GCS artifacts (checkpoint step+ts for training, trial count / output-bucket population for datagen). NEVER diagnose liveness from raw `iris job logs` tail (interleaved multi-rank lines; --no-tail returns STARTUP lines not latest; a clean preempt emits no terminal log line). See memory iris_liveness_tooling_not_logtail.

(A) SWEEP both iris clusters (marin GCP + cw-us-east-02a) and the native route (expect 401). Note Daytona snapshot count on the cli org. STANDING CLEANUP: each tick, proactively reclaim idle harbor__ snapshots >120min via /Users/benjaminfeuer/Documents/OpenThoughts-Agent/scripts/daytona/daytona_snapshot_manager.py --api-key-env DAYTONA_API_KEY --stale-days 0.0833 --delete-stale --yes (run from the OT-Agent dir, secrets sourced). Deletes ONLY idle harbor__ env snapshots; the --name-prefix harbor__ default GUARDS base images (daytonaio/sandbox:*, daytona-*, windows-*) — NEVER delete those, never ACTIVE-recent (<120min). Report before/after count.

(B) CAMPAIGN qwen3.5-122b-131k-datagen-opencode-iris — keep-3 steady-state. Box table (arm | dataset | completed/total | Mean | liveness). Confirm each arm RUNNING+serving+advancing (advancing = GCS trial count up vs last tick), single serve dir, no job.py:263.
  MEAN COLUMN IS MANDATORY — pull it from the harbor `<done>/<total> Mean: <X>` progress line in the job logs: `/Users/benjaminfeuer/Documents/marin/.venv/bin/iris --cluster=marin job logs /benjaminfeuer/<job> 2>&1 | grep -aoE '[0-9]+/[0-9]+ Mean: [-0-9.]+' | tail -1` (retry on a transient finelog `dns error`/`StatsError`; that same line ALSO gives the freshest completed/total, more current than result.json). This targeted metric grep is NOT the prohibited liveness-by-log-tail — it is REQUIRED every tick. Mark `—` ONLY when the arm genuinely emits no Mean line (no verifier); do NOT default to `—` because result.json lacks a mean field. (result.json carries no mean — that is expected and is not a reason to drop the column.)
  >=95% KILL-AND-HARVEST (operator directive 2026-07-14): if a state-3 RUNNING arm is past 95% completed/total on ANY tick, KILL the child (autonomously authorized; child only) and HARVEST it as a terminal arm right now — do NOT await the last stragglers (the long tail is not worth the held v5p-8). Treat it exactly like a >=60% terminal arm: rescue via datagen-job-cleanup + refill. (Applies to normal deterministic-verifier arms; a still-cold-compiling or resume-scanning arm that only shows 95% because its total is not yet known is exempt — require real advancing trials.)
  TERMINAL arm (state 4/5/6): if >=60% complete, dispatch a subagent ARMED WITH /Users/benjaminfeuer/Documents/OpenThoughts-Agent/.agents/skills/datagen-job-cleanup (read that SKILL.md, follow its idempotent steps): gs:// rescue of the OUTER recorded output dir (resolve via hpc.iris.job_output_resolver — single-region gs://marin-<region>/ot-agent/<job> for new jobs, multi-region gs://marin-models-{us,eu}/ for legacy; NEVER hardcode) so logs/ literals ride along -> avg_turns realness check -> HF upload via make_and_upload_trace_dataset.py --episodes last with literals AUTO-INCLUDED and --served_model Qwen/Qwen3.5-122B-A10B-FP8 -> verify HF non-empty + Literal yield X/Y (X>0) + count_populated_literal_rows>0 -> report rows+yield. (If <60%, resume instead.) THEN submit a refill (next un-launched tracker dataset) on the newest validated :tpu image --preemptible (no --gcs-output-dir, so the single-region region-pin engages) and let iris schedule; update the tracker with the cleanup (repo+rows) and the refill.
  CONFIRMED-WEDGED datagen (state 3 RUNNING but authoritative-state RUNNING AND 0 trials / empty output bucket for hours with no engine-serving marker, OR harbor frozen >=3h fd-monitor-only per datagen-job-cleanup) -> kill the child + refill (autonomously authorized; child only). Distinguish from a healthy cold-compile/resume-scan (engine bringing up / recompiling) — do NOT kill those.
  If keep-N<3 for any reason, submit refill(s), let iris schedule — do NOT gate on a capacity guess. Preemptible jobs stay preemptible. The LLM-judge-verified datasets (laion/stackexchange-superuser-sandboxes-verified, laion/stackexchange-tezos-sandboxes-verified — tracker rows 120/121) RUN IN THE NORMAL keep-3 SEQUENCE — do NOT skip/hold them: their per-task task.toml propagates OPENAI_API_KEY into the trial verifier sandbox and the standard launch already passes --secrets-env "$DC_AGENT_SECRET_ENV" (which carries OPENAI_API_KEY), so the litellm judge scores real rewards. See datagen-launch-iris (Prerequisites > LLM-judge datasets) for the mechanism.

(C) MIDTRAIN 1e23_p33m67_k0p20 (you OWN this, monitor-only) — ONE-LINE STATUS via iris_ops.py/SQL (NOT log-tail): child state + live step / newest TEMP checkpoint step+ts (gs://marin-us-east5/tmp/ttl=14d/checkpoints-temp/.../delphi-1e23-p33m67-k0p20-lr0.67-b6607e/checkpoints/step-*/metadata.json) + whether the checkpoint step ADVANCED since last tick. NO memory-bounded field (cgroup tracking RETIRED 2026-07-13). A PENDING/re-placing child (preempt/crash-teardown/tier-monotonicity/capacity) is NORMAL — state it, do NOT flag/escalate/bounce; a stale checkpoint under preempt churn is expected. ONLY act on: a confirmed state-3 RUNNING-but-frozen wedge (authoritative state RUNNING + checkpoint stalled across multiple ticks + pod mismatch) -> Option A bounce the WEDGED CHILD only (never coordinator); or a confirmed cgroup OOM.

Relay a tight A/B/C report (campaign as a box-drawing datagen table with Mean).

If you change the cadence or scope, update BOTH the cron/prompt above and the live job (delete + recreate), and keep monitor-cron-sweep-iris in sync — so this skill stays the canonical copy.