Run any Skill in Manus with one click

monitor-restore-unified

Stars260

Forks36

UpdatedJuly 15, 2026 at 14:47

Re-register the every-3-hours UNIFIED OPS TICK cron — the CURRENT operator-owned monitor for the qwen3.5-122b-131k-datagen-opencode campaign (keep-3 datagen with autonomous rescue+refill) AND the delphi midtrain 1e23_p33m67_k0p20 (monitor-only), plus a marin+CoreWeave sweep and standing Daytona snapshot cleanup. The cron is session-only and recurring crons auto-expire after 7 days, so it's routinely lost on a session restart. Use at the start of a new session, after a disconnect/restart, or when the user asks to restore the monitor cron. Supersedes monitor-restore-iris (which is the OLDER datagen-OUT-of-scope variant); use THIS skill when the current session owns the 131k datagen campaign + midtrain. The per-tick sweep methodology lives in monitor-cron-sweep-iris; the table format in monitor-job-tables.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Run Skill in Manus

Source

open-thoughts

open-thoughts/OpenThoughts-Agent

View GitHub Repository View Creator Repositories

Download

Run Skill in Manus

Related occupationsSOC

Based on SOC occupation classification

Network and Computer Systems AdministratorsComputer and Mathematical Occupations·SOC 15-1244

SKILL.md

readonly

More from this repository

same repository

rl-job-health-deep-dive

open-thoughts/OpenThoughts-Agent

Deep single-RL-job health probe → a KILL / NO-KILL / ERROR recommendation for the supervisor. Dispatched as a subagent on every monitor tick for RL jobs in NEW/UNTESTED settings (new config/geometry/model, "debug" or "smoke-test" flavor, first launches after a code/config change), and whenever a running job looks starved or wedged. Goes BEYOND state-poll + table metrics: captures the job's logs + trace_jobs + GPU view, then runs four gates — (A) liveness, (B) resource utilization / engine subscription, (C) rollout quality — and emits ONE evidence-backed verdict. The subagent NEVER kills — it recommends; the supervisor owns the kill. This skill holds the METHODOLOGY only; every cluster-access and codebase fact is a POINTER into .claude/ops/<cluster>/ and .claude/projects/<dep>/ (those are the single source of truth — do not re-encode them here, they go stale).

2026-07-15260

supervisor-init

open-thoughts/OpenThoughts-Agent

Bootstrap the supervisor role at the start of a session — the human-facing lab supervisor who manages a large multi-experiment ML-ops operation: the single interface between the user and subagents / crons / top-level tools, keeper of secrets, and tracker of everything in flight. Run this FIRST in a fresh session (or when the user says "set up", "init", "take over", "you're the supervisor", "get oriented"). It walks an init checklist (orient in .claude, load the local env, take custody of secrets, survey in-flight work + crons + subagents), states the operating discipline (verify subagent work, fix proactively without gating unless needed, secrets only via env vars), and concludes by (re)creating the 3-hour sweep loop (monitor-restore) and running an initial sweep (monitor-cron-sweep).

2026-07-14260

datagen-job-cleanup

open-thoughts/OpenThoughts-Agent

Post-run cleanup for a datagen (trace-generation) job on an HPC cluster (Jupiter/Leonardo/Perlmutter): get the generated traces onto HF (penfever org) and free disk. There is NO model checkpoint — the artifact is the trace dataset. Covers the TIMEOUT-strands-traces gotcha (uploads silently never run), the ONE-level trace_jobs nesting (vs RL's double-nest), the real-vs-failed sanity check (avg_turns ≈ 1.0 = dead run, don't upload), the otagent-env uploader, the non-empty HF verify, and safe disk cleanup (one-off task dirs vs shared canonical tasks; leave Daytona snapshots). Use when a datagen/trace job finishes (COMPLETED or TIMEOUT) and its traces need uploading + verifying, or when consolidating a chunked datagen run. Distinct from RL/SFT cleanup (those publish a model checkpoint).

2026-07-13260

datagen-launch-iris

open-thoughts/OpenThoughts-Agent

Launch, monitor, and manually clean up a trajectory-generation (datagen) job on Marin's Iris TPU cluster via the OpenThoughts-Agent entrypoint. Use when asked to start, watch, rescue, or kill a datagen/tracegen run on Iris.

2026-07-13260

eval-agentic-cleanup

open-thoughts/OpenThoughts-Agent

Audit + recover a finished agentic eval. ALWAYS start with the read-only, idempotent completeness/health audit (§0): job finished? score present + non-zero + not obviously broken? HF traces present + linked? trial count ≈ n_rep × benchmark_size? — it writes nothing and recommends an action per check. Then run only the flagged remediations: manual HF-trace upload + Supabase DB-registration via manual_db_eval_push.py, the vLLM-numeric-ID → real-HF-model-name fix (with cross-user FK safety), verify, free disk. Use to verify an eval is truly complete, when a sweep finds an eval that didn't upload/register or has a broken/zero score or short trial count, or to re-register/correct an eval's model/traces. Distinct from the model- publishing cleanups (rl-agentic-job-cleanup / sft-job-cleanup) and datagen-job-cleanup — this is the EVAL path.

2026-07-13260

eval-agentic-launch-iris

open-thoughts/OpenThoughts-Agent

Launch, monitor, and manually clean up an eval job on Marin's Iris TPU or CoreWeave H100x8 GPU cluster via the OpenThoughts-Agent entrypoint. Use when asked to start, watch, or kill a model evaluation (evalchemy / agent-harness benchmarks) on Iris.

2026-07-13260

name

monitor-restore-unified

description

monitor-restore-unified

📍 Iris orientation — read first. Before acting, read the Iris tools catalog (.claude/ops/iris/iris_tools.md) and the Iris ops directory (.claude/ops/iris/ — iris_job_lifecycle.md for the TPU marin particulars, coreweave_gpu_ops.md for GPU). They carry the binding access/preamble/gotchas and the helper-script inventory below.

The recurring UNIFIED OPS TICK cron is session-only (lives in the Claude session, not on disk) and recurring crons auto-expire after 7 days — so it is routinely lost on a session restart. This skill is the durable source of truth for re-creating it: the canonical cron prompt below is what gets (re-)installed — copy it verbatim into CronCreate. The per-tick sweep methodology is monitor-cron-sweep-iris; the table format is monitor-job-tables.

Scope — how this differs from monitor-restore-iris

monitor-restore-iris installs the OLDER prompt where datagen is OUT of scope (a different agent owned it; that monitor covered eval + Levanter training + GPU-RL only). This skill is the opposite: the current session OWNS

(B) the qwen3.5-122b-131k-datagen-opencode-iris campaign — keep-3 steady-state, with autonomous TERMINAL rescue (→ HF with literals) + refill, and confirmed-wedged kill+refill.
(C) the delphi midtrain 1e23_p33m67_k0p20 — monitor-only.
(A) the marin+CoreWeave sweep + native-route check + standing Daytona cleanup.

Install whichever matches who owns datagen this session. If both are somehow live, they conflict on datagen — keep only one (delete the other via CronDelete).

When to run

Start of a new session where the 131k datagen campaign / midtrain are in flight.
The user says the monitor/cron is gone, down, "not firing," or after a disconnect.
After ~7 days (expiry).

Steps

Check if it already exists — call CronList. If a recurring job whose prompt begins "UNIFIED OPS TICK" is present, do nothing (a duplicate causes redundant SQL/tunnel load). If a stale variant (e.g. the datagen-out-of-scope monitor-restore-iris prompt) is present and this session owns datagen, CronDelete it and install the prompt below.
If absent, call CronCreate with:
- cron: 23 */3 * * * (every 3 h at :23 — off the :00/:30 marks)
- recurring: true
- prompt: the exact text in the fenced block below.
Tell the user the new job id + the two caveats: session-only (dies when this Claude session exits — re-run this skill next session) and 7-day auto-expiry.

Notes

durable: true is NOT honored in this harness (still session-only) — this skill IS the persistence layer.
The cron only fires while the REPL is idle (not mid-task).
Liveness tooling (operator directive 2026-07-13, memory [[iris_liveness_tooling_not_logtail]]): judge liveness/wedge via scripts/iris/watch_job_state.py + direct iris SQL, and progress via GCS artifacts (checkpoint step+ts for training, trial count / output-bucket population for datagen). NEVER diagnose liveness from raw iris job logs tail (interleaved multi-rank lines; --no-tail returns startup lines; a clean preempt emits no terminal log line).
cgroup memory tracking is RETIRED (2026-07-13): midtrain (C) reports child state + checkpoint-step delta only — no memory-bounded field.
Single-region output migration (OT-Agent c76dd23a): new datagen launches pass NO --gcs-output-dir, so the launcher's region-pin routes output to a co-located single-region bucket (gs://marin-us-<region>/ot-agent/<job>). Rescue resolves each job's RECORDED output URI via hpc.iris.job_output_resolver — legacy jobs stay on multi-region marin-models-{us,eu}. NEVER hardcode an output bucket.
Companion skills: monitor-cron-sweep-iris (the tick methodology), monitor-job-tables (the datagen box table WITH Mean column), datagen-job-cleanup (canonical idempotent rescue for a TERMINAL/wedged arm — dispatch a subagent armed with it), datagen-launch-iris (refill).

Canonical cron prompt (copy verbatim into CronCreate)

UNIFIED OPS TICK. Source /Users/benjaminfeuer/Documents/secrets.env (never echo secrets). Spawn a general-purpose agent that reads the absolute-path ops docs and reports back; use PATH /Users/benjaminfeuer/miniconda3/envs/otagent/bin for iris+python. Wrap cw kubectl and any recursive gsutil in timeout. Use ABSOLUTE paths everywhere. Format the campaign status as a DATAGEN box-drawing table WITH a Mean-reward column per /Users/benjaminfeuer/Documents/OpenThoughts-Agent/.claude/skills/monitor-job-tables.

LIVENESS TOOLING (operator directive 2026-07-13): judge job liveness/wedge via /Users/benjaminfeuer/Documents/OpenThoughts-Agent/scripts/iris/watch_job_state.py + direct iris SQL query, and progress via GCS artifacts (checkpoint step+ts for training, trial count / output-bucket population for datagen). NEVER diagnose liveness from raw `iris job logs` tail (interleaved multi-rank lines; --no-tail returns STARTUP lines not latest; a clean preempt emits no terminal log line). See memory iris_liveness_tooling_not_logtail.

(A) SWEEP both iris clusters (marin GCP + cw-us-east-02a) and the native route (expect 401). Note Daytona snapshot count on the cli org. STANDING CLEANUP: each tick, proactively reclaim idle harbor__ snapshots >120min via /Users/benjaminfeuer/Documents/OpenThoughts-Agent/scripts/daytona/daytona_snapshot_manager.py --api-key-env DAYTONA_API_KEY --stale-days 0.0833 --delete-stale --yes (run from the OT-Agent dir, secrets sourced). Deletes ONLY idle harbor__ env snapshots; the --name-prefix harbor__ default GUARDS base images (daytonaio/sandbox:*, daytona-*, windows-*) — NEVER delete those, never ACTIVE-recent (<120min). Report before/after count.

(B) CAMPAIGN qwen3.5-122b-131k-datagen-opencode-iris — keep-3 steady-state. Box table (arm | dataset | completed/total | Mean | liveness). Confirm each arm RUNNING+serving+advancing (advancing = GCS trial count up vs last tick), single serve dir, no job.py:263.
  >=95% KILL-AND-HARVEST (operator directive 2026-07-14): if a state-3 RUNNING arm is past 95% completed/total on ANY tick, KILL the child (autonomously authorized; child only) and HARVEST it as a terminal arm right now — do NOT await the last stragglers (the long tail is not worth the held v5p-8). Treat it exactly like a >=60% terminal arm: rescue via datagen-job-cleanup + refill. (Applies to normal deterministic-verifier arms; a still-cold-compiling or resume-scanning arm that only shows 95% because its total is not yet known is exempt — require real advancing trials.)
  TERMINAL arm (state 4/5/6): if >=60% complete, dispatch a subagent ARMED WITH /Users/benjaminfeuer/Documents/OpenThoughts-Agent/.claude/skills/datagen-job-cleanup (read that SKILL.md, follow its idempotent steps): gs:// rescue of the OUTER recorded output dir (resolve via hpc.iris.job_output_resolver — single-region gs://marin-<region>/ot-agent/<job> for new jobs, multi-region gs://marin-models-{us,eu}/ for legacy; NEVER hardcode) so logs/ literals ride along -> avg_turns realness check -> HF upload via make_and_upload_trace_dataset.py --episodes last with literals AUTO-INCLUDED and --served_model Qwen/Qwen3.5-122B-A10B-FP8 -> verify HF non-empty + Literal yield X/Y (X>0) + count_populated_literal_rows>0 -> report rows+yield. (If <60%, resume instead.) THEN submit a refill (next un-launched tracker dataset) on the newest validated :tpu image --preemptible (no --gcs-output-dir, so the single-region region-pin engages) and let iris schedule; update the tracker with the cleanup (repo+rows) and the refill.
  CONFIRMED-WEDGED datagen (state 3 RUNNING but authoritative-state RUNNING AND 0 trials / empty output bucket for hours with no engine-serving marker, OR harbor frozen >=3h fd-monitor-only per datagen-job-cleanup) -> kill the child + refill (autonomously authorized; child only). Distinguish from a healthy cold-compile/resume-scan (engine bringing up / recompiling) — do NOT kill those.
  If keep-N<3 for any reason, submit refill(s), let iris schedule — do NOT gate on a capacity guess. Preemptible jobs stay preemptible.

(C) MIDTRAIN 1e23_p33m67_k0p20 (you OWN this, monitor-only) — ONE-LINE STATUS via watch_job_state.py/SQL (NOT log-tail): child state + live step / newest TEMP checkpoint step+ts (gs://marin-us-east5/tmp/ttl=14d/checkpoints-temp/.../delphi-1e23-p33m67-k0p20-lr0.67-b6607e/checkpoints/step-*/metadata.json) + whether the checkpoint step ADVANCED since last tick. NO memory-bounded field (cgroup tracking RETIRED 2026-07-13). A PENDING/re-placing child (preempt/crash-teardown/tier-monotonicity/capacity) is NORMAL — state it, do NOT flag/escalate/bounce; a stale checkpoint under preempt churn is expected. ONLY act on: a confirmed state-3 RUNNING-but-frozen wedge (authoritative state RUNNING + checkpoint stalled across multiple ticks + pod mismatch) -> Option A bounce the WEDGED CHILD only (never coordinator); or a confirmed cgroup OOM.

Relay a tight A/B/C report (campaign as a box-drawing datagen table with Mean).

If you change the cadence or scope, update BOTH the cron/prompt above and the live job (delete + recreate), and keep monitor-cron-sweep-iris (the procedure) in sync — so this skill stays the canonical copy.