Re-register the every-3-hours UNIFIED OPS TICK cron — the CURRENT operator-owned monitor for the qwen3.5-122b-131k-datagen-opencode campaign (keep-3 datagen with autonomous rescue+refill) AND the delphi midtrain 1e23_p33m67_k0p20 (monitor-only), plus a marin+CoreWeave sweep and standing Daytona snapshot cleanup. The cron is session-only and recurring crons auto-expire after 7 days, so it's routinely lost on a session restart. Use at the start of a new session, after a disconnect/restart, or when the user asks to restore the monitor cron. Supersedes monitor-restore-iris (which is the OLDER datagen-OUT-of-scope variant); use THIS skill when the current session owns the 131k datagen campaign + midtrain. The per-tick sweep methodology lives in monitor-cron-sweep-iris; the table format in monitor-job-tables.
Deep single-RL-job health probe → a KILL / NO-KILL / ERROR recommendation for the supervisor. Dispatched as a subagent on every monitor tick for RL jobs in NEW/UNTESTED settings (new config/geometry/model, "debug" or "smoke-test" flavor, first launches after a code/config change), and whenever a running job looks starved or wedged. Goes BEYOND state-poll + table metrics: captures the job's logs + trace_jobs + GPU view, then runs four gates — (A) liveness, (B) resource utilization / engine subscription, (C) rollout quality — and emits ONE evidence-backed verdict. The subagent NEVER kills — it recommends; the supervisor owns the kill. This skill holds the METHODOLOGY only; every cluster-access and codebase fact is a POINTER into .claude/ops/<cluster>/ and .claude/projects/<dep>/ (those are the single source of truth — do not re-encode them here, they go stale).
Bootstrap the supervisor role at the start of a session — the human-facing lab supervisor who manages a large multi-experiment ML-ops operation: the single interface between the user and subagents / crons / top-level tools, keeper of secrets, and tracker of everything in flight. Run this FIRST in a fresh session (or when the user says "set up", "init", "take over", "you're the supervisor", "get oriented"). It walks an init checklist (orient in .claude, load the local env, take custody of secrets, survey in-flight work + crons + subagents), states the operating discipline (verify subagent work, fix proactively without gating unless needed, secrets only via env vars), and concludes by (re)creating the 3-hour sweep loop (monitor-restore) and running an initial sweep (monitor-cron-sweep).
Post-run cleanup for a datagen (trace-generation) job on an HPC cluster (Jupiter/Leonardo/Perlmutter): get the generated traces onto HF (penfever org) and free disk. There is NO model checkpoint — the artifact is the trace dataset. Covers the TIMEOUT-strands-traces gotcha (uploads silently never run), the ONE-level trace_jobs nesting (vs RL's double-nest), the real-vs-failed sanity check (avg_turns ≈ 1.0 = dead run, don't upload), the otagent-env uploader, the non-empty HF verify, and safe disk cleanup (one-off task dirs vs shared canonical tasks; leave Daytona snapshots). Use when a datagen/trace job finishes (COMPLETED or TIMEOUT) and its traces need uploading + verifying, or when consolidating a chunked datagen run. Distinct from RL/SFT cleanup (those publish a model checkpoint).
Launch, monitor, and manually clean up a trajectory-generation (datagen) job on Marin's Iris TPU cluster via the OpenThoughts-Agent entrypoint. Use when asked to start, watch, rescue, or kill a datagen/tracegen run on Iris.
Audit + recover a finished agentic eval. ALWAYS start with the read-only, idempotent completeness/health audit (§0): job finished? score present + non-zero + not obviously broken? HF traces present + linked? trial count ≈ n_rep × benchmark_size? — it writes nothing and recommends an action per check. Then run only the flagged remediations: manual HF-trace upload + Supabase DB-registration via manual_db_eval_push.py, the vLLM-numeric-ID → real-HF-model-name fix (with cross-user FK safety), verify, free disk. Use to verify an eval is truly complete, when a sweep finds an eval that didn't upload/register or has a broken/zero score or short trial count, or to re-register/correct an eval's model/traces. Distinct from the model- publishing cleanups (rl-agentic-job-cleanup / sft-job-cleanup) and datagen-job-cleanup — this is the EVAL path.
Launch, monitor, and manually clean up an eval job on Marin's Iris TPU or CoreWeave H100x8 GPU cluster via the OpenThoughts-Agent entrypoint. Use when asked to start, watch, or kill a model evaluation (evalchemy / agent-harness benchmarks) on Iris.
The PROCEDURE for one every-3-hours Iris job-status sweep — primarily the marin TPU datagen/eval jobs ("iris" here = the marin TPU cluster), plus CoreWeave GPU-RL as monitor-only. Query both clusters, run the harbor analyzer on each active TPU datagen/eval job, classify every job (datagen / eval / other / GPU-RL) and apply its treatment, then take the standing DATAGEN-ONLY autonomous actions (auto-rescue, keep-2-in-flight). This is the methodology the recurring cron prompt runs; the cron itself is (re)installed via monitor-restore-iris. Use for "run an iris sweep / cluster sweep now" or as the reference behind each cron tick.