OpenThoughts-Agent

The PROCEDURE for one every-3-hours Iris job-status sweep — primarily the marin TPU datagen/eval jobs ("iris" here = the marin TPU cluster), plus CoreWeave GPU-RL as monitor-only. Query both clusters, run the harbor analyzer on each active TPU datagen/eval job, classify every job (datagen / eval / other / GPU-RL) and apply its treatment, then take the standing DATAGEN-ONLY autonomous actions (auto-rescue, keep-2-in-flight). This is the methodology the recurring cron prompt runs; the cron itself is (re)installed via monitor-restore-iris. Use for "run an iris sweep / cluster sweep now" or as the reference behind each cron tick.

Launch, monitor, and manually clean up a trajectory-generation (datagen) job on Marin's Iris TPU cluster via the OpenThoughts-Agent entrypoint. Use when asked to start, watch, rescue, or kill a datagen/tracegen run on Iris.

Launch a datagen (trace-generation) job on an HPC cluster (Jupiter/Leonardo/Perlmutter) via the OpenThoughts-Agent `hpc.launch --job_type datagen` entrypoint — the cluster-AGNOSTIC general flow: extract tasks from a parquet, then submit a managed vLLM-serve + Harbor/Daytona trace run that uploads the trajectories to an HF repo. Use when asked to start a datagen / trace-generation job, generate agent traces from a task dataset, or advance / start a row in the MiniMax-M2.7 datagen tracker. For the Iris TPU path use **datagen-launch-iris** instead; per-cluster particulars (ssh, paths, conda env, which vllm-serve config matches the cluster's GPUs, JSC pre-download) live in `.claude/ops/<cluster>/`.

Reclaim idle Daytona SNAPSHOTS org-wide to free space under the 60-snapshot cap, using scripts/daytona/daytona_snapshot_manager.py. Deletes only harbor__ per- environment snapshots idle past the standing threshold GT'd in .claude/projects/daytona/daytona.md — NEVER the shared base/template images (daytonaio/sandbox:*, daytona-*, windows-*), which do not rebuild-on-demand. Use when a datagen/eval launch hits SnapshotCapExceeded, when a monitor sweep finds the cap full, or as routine cap hygiene. This is the org-wide RECLAIM tool; to shrink a SINGLE dataset's unique-environment count instead, use datagen-reduce-dataset-snapshots. Snapshots are a DIFFERENT resource from sandboxes (for sandbox cleanup use utils-cleanup-stale-sandboxes).

Re-register the every-3-hours Iris job-monitor cron (status check + datagen auto-rescue/keep-2-in-flight) if it has been lost. Primarily the marin TPU datagen/eval jobs ("iris" = the marin TPU cluster); also queries CoreWeave (cw-us-east-02a) GPU-RL as monitor-only. The cron is session-only and recurring crons auto-expire after 7 days, so it's routinely lost on a session restart. Use at the start of a new session, after a restart, or when the user asks to restore/check the iris monitoring cron. The sweep PROCEDURE the cron runs lives in monitor-cron-sweep-iris.

Launch / relaunch agentic RL (SkyRL terminal_bench + Harbor + Daytona) on JSC Jupiter (GH200). Covers the dense 8B/32B FSDP2 arms (seqnorm, TIS, shaped, symclip, lrboost, loopshape) and the MoE/80B Megatron arms (Qwen3-Coder-30B-A3B, Qwen3-Next-80B-A3B) — the exact `python -m hpc.launch --job_type rl` flag set, which flags vary per arm (config / model_path / train_data / num_nodes), runtime+SIF selection, the Daytona RL-org + chain-restart conventions, and the standing constraints (≤6 RL/cluster, a3 CONCLUDED, TIMEOUT restarts are normal). Use when asked to launch / relaunch / refill an agentic SkyRL RL run on Jupiter. Reference: notes/ot-agent/rl_experiments.md, .claude/ops/jupiter/{ops.md,ENVIRONMENT_MAP.md}.

Bootstrap the supervisor role at the start of a session — the human-facing lab supervisor who manages a large multi-experiment ML-ops operation: the single interface between the user and subagents / crons / top-level tools, keeper of secrets, and tracker of everything in flight. Run this FIRST in a fresh session (or when the user says "set up", "init", "take over", "you're the supervisor", "get oriented"). It walks an init checklist (orient in .claude, load the local env, take custody of secrets, survey in-flight work + crons + subagents), states the operating discipline (verify subagent work, fix proactively without gating unless needed, secrets only via env vars), and concludes by (re)creating the 3-hour sweep loop (monitor-restore) and running an initial sweep (monitor-cron-sweep).

Produce a comprehensive cross-cluster job-status update for a recurring N-hourly cluster sweep. Gather squeue/sacct on each cluster (validating against false-drain), bucket every active + recently-terminated job by type (RL / SFT / datagen / eval / catch-all), pull each type's signals, render them in the job_monitor_table.md formats, and flag completions (→ the matching cleanup skill), genuine failures (→ diagnose + agent_logs), and per-type health red-flags. Cluster-AGNOSTIC — ssh strings, code/log/exp paths, concurrency caps, gpu-mem ceilings live in `.claude/ops/<cluster>/`. Use for "run a cluster sweep", the N-hourly cron, or "give me a status update on all jobs".

Re-create the local 3-hour tri-cluster cluster-sweep loop (Leonardo + CoreWeave(iris) + TACC(Vista); Jupiter SKIPPED until ~Jul 12) — the autonomous ML-ops monitor — if it has been lost. The loop is session-only and is dropped on any session restart, so re-establish it at the start of a new session or whenever the user asks to restore/restart the 3h sweep/cron/monitor. Sets a /loop 3h (or equivalent recurring cron) whose task is the canonical sweep prompt below: status-table active/pending/completed jobs, auto-cleanup+DB-register completions, diagnose+remediate failures via subagents, log to agent_logs + claude_experiments.md. The prompt block here is the source of truth — copy it verbatim.

Safely purge stale, never-populated `sandbox_jobs` placeholder rows (eval launches that died/stalled before scoring) from the OT-Agent Supabase registry. Removes ONLY dead `Pending`/`Started` rows WE OWN that are >36h old with null `metrics`/`stats`/`ended_at`, via the mandatory cross-user FK-safety pre-check + the REQUIRED grandchild→child→job cascade delete (`sandbox_trial_model_usage` → `sandbox_trials` → `sandbox_jobs`). Other users' stale rows are REPORTED, never deleted. DRY-RUN first, then delete, then re-read. Use when the registry is clogged with dead placeholder eval rows, when a sweep flags stale Pending/Started/"Running" eval entries, or when the eval listener's dedup is mis-firing on dead rows. The general CRUD/read/aggregation skill is `crud-otagent-supabase`.

Format HPC job-status reports as box-drawing tables, bucketed by job type (RL · SFT · Datagen · Eval · Catch-all), with the right metric columns, signal thresholds, and red-flags per bucket. Use whenever reporting active/recently-terminated job status — during a cron sweep (driven by monitor-cron-sweep), an ad-hoc "how are my jobs doing", or a single-job progress update. Covers which metrics are mandatory (entropy + collapse signals for RL, not just step/reward/grad), where to pull live status (SFT .out vs trainer_log.jsonl), the RL collapse-warning rule, and which log lines are benign noise vs real faults (shm_broadcast 600s, rollout_train_prob_diff_mean millions). Cluster-agnostic — refer to .claude/ops for paths.

Given a list of models (HF name stubs) that have valid agentic ID eval scores in Supabase, build a ranking table: raw per-benchmark accuracy on the 3 ID benchmarks (SWE-Bench-100, OT-TBLite=dev_set_v2, Terminal-Bench-2.0=tb2), HF links to each eval's trace dataset, and a NORMALIZED column = average per-benchmark z-score, ranked. Normalization matches the OpenThoughts-Agent paper (otagent-paper/02_arXiv/otagent.tex §Pipeline): per-benchmark z over the candidate set, averaged. Read-only. Use when asked to rank models / ablation arms by their ID evals the way the paper does.

Run the Iris harbor job-history analyzer (scripts/iris/analyze_job_history.py) on a datagen/eval job and read its JSON sidecar for trustworthy throughput / preemption / productive-trial stats. Use whenever a status check needs REAL metrics (gen tok/s, cycles, non_empty rate, harbor exceptions) instead of an eyeballed log tail. It now queries the finelog log store directly (live ∪ GCS, deduped) — FAST (seconds, not minutes) and it ASSERTS completeness across all preempted attempts/generations, failing loud rather than returning fragments.

Guardrailed DELETE of auto-registered eval `sandbox_jobs` rows that DID score but FAILED the harvest gate — partial/contaminated evals (valid-complete <90%, or infra-error >10%, or AgentTimeout ≥80%). These are the "DE-REGISTER candidate" rows the flawed_summ harvest defers. DISTINCT from `crud-purge-stale-eval-placeholders` (that removes NEVER-populated Pending/Started rows; this removes rows that populated stats/metrics but are below-gate). Use when asked to remove "partial / below-gate / <90%-completed-without-errors" evals owned by us. The MANDATORY parts: the AgentTimeout-is-BENIGN gate (a literal n_errors formula is catastrophically wrong), the cross-user FK-safety pre-check, and the grandchild→child→job cascade.

Post-run cleanup for a datagen (trace-generation) job on an HPC cluster (Jupiter/Leonardo/Perlmutter): get the generated traces onto HF (penfever org) and free disk. There is NO model checkpoint — the artifact is the trace dataset. Covers the TIMEOUT-strands-traces gotcha (uploads silently never run), the ONE-level trace_jobs nesting (vs RL's double-nest), the real-vs-failed sanity check (avg_turns ≈ 1.0 = dead run, don't upload), the otagent-env uploader, the non-empty HF verify, and safe disk cleanup (one-off task dirs vs shared canonical tasks; leave Daytona snapshots). Use when a datagen/trace job finishes (COMPLETED or TIMEOUT) and its traces need uploading + verifying, or when consolidating a chunked datagen run. Distinct from RL/SFT cleanup (those publish a model checkpoint).

Audit + recover a finished agentic eval. ALWAYS start with the read-only, idempotent completeness/health audit (§0): job finished? score present + non-zero + not obviously broken? HF traces present + linked? trial count ≈ n_rep × benchmark_size? — it writes nothing and recommends an action per check. Then run only the flagged remediations: manual HF-trace upload + Supabase DB-registration via manual_db_eval_push.py, the vLLM-numeric-ID → real-HF-model-name fix (with cross-user FK safety), verify, free disk. Use to verify an eval is truly complete, when a sweep finds an eval that didn't upload/register or has a broken/zero score or short trial count, or to re-register/correct an eval's model/traces. Distinct from the model- publishing cleanups (rl-agentic-job-cleanup / sft-job-cleanup) and datagen-job-cleanup — this is the EVAL path.

Launch, monitor, and manually clean up an eval job on Marin's Iris TPU or CoreWeave H100x8 GPU cluster via the OpenThoughts-Agent entrypoint. Use when asked to start, watch, or kill a model evaluation (evalchemy / agent-harness benchmarks) on Iris.

Launch / relaunch agentic MarinSkyRL (SkyRL GRPO) RL on Marin's Iris / CoreWeave GPU cluster (cw-us-east-02a, 8x H100-80GB + InfiniBand per node) via `python -m rl.cloud.launch_rl_iris` + the gpu-rl Docker image (NO Apptainer SIF). Covers the dense 8B FSDP2 arms (seqnorm + TIS) and the MoE 30B-A3B arms (CP + DCP=2 + R3 @ 131k) — the exact launcher flag set (`--rl_config`, `--model_path`, `--train_data`, `--num-nodes`, `--rendezvous-dir`, `--job-name`, `--priority`, `--cpu`, `--max-retries`), the gang/leafgroup/Kueue multi-node Ray rendezvous, the iris config-authoring rules (NO container block, load-bearing top-level `extra_env:` forwarding, disaggregated placement + explicit `num_inference_engines`, SIF→Docker env translation), and the bring-up gotchas learned this week (`--cpu 48`, `--max-retries ≥1`). Use when asked to launch / relaunch an agentic SkyRL RL run on Iris / CoreWeave. Cluster access/hardware particulars live in .claude/ops/iris/coreweave_gpu_ops.md (this skill defers to it). Reference: rl/clou

Deep single-RL-job health probe → a KILL / NO-KILL recommendation for the supervisor. Dispatched as a subagent on every monitor tick for RL jobs in NEW/UNTESTED settings (new config/geometry/model, "debug" or "smoke-test" flavor, first launches after a code/config change). Goes BEYOND state-poll + the table metrics: syncs the job's trace_jobs + stderr/stdout + Ray logs to ~/Documents/experiments/traces via the EXISTING capture tool (CoreWeave: scripts/iris/peek_rl_rollouts.sh `pull`), then runs four gates — (1) liveness (tail stdout/stderr: zombie/wedged/dead?), (2) resource utilization (live-poll GPUs: all inference engines alive + generating at a hardware/model-size-reasonable cadence per the serving LUT; training stage not VRAM/RAM-OOM), (3) rollout quality (trace_jobs: trials init/complete, non-zero rewards, turns completing, agent outputs sane, tasks hard, verifiers firing), and emits ONE verdict with evidence + next steps. The subagent NEVER kills — it recommends; the supervisor owns the kill (standing

Build a clean per-dataset summary table/CSV for a datagen (trajectory-generation) campaign — one row per task source with Status (COMPLETED / FAILED / RUNNING / NOT STARTED), N Trials Completed, Mean Turns/Trace, Mean Tok/Trace, Mean Reward, and the HF trace-repo link. Use when asked to "summarize the campaign", "which datasets did we complete + their rewards/trials", "build a completion table/CSV", or to reconcile a prose tracker into auditable per-dataset metrics. Computes metrics by STREAMING each uploaded HF trace dataset (disk-bounded) and reusing the canonical OT-Agent analysis tools (scripts/analysis/utils.py: extract_conversation_text / count_turns / extract_reward) + the Qwen3-8B tokenizer; HF-ground-truths Status by probing each repo. Reference impl: scripts/analysis/build_campaign_summary_csv.py. Related skills: analyze-dataset-token-length (token method), analyze-job-history-iris (harbor Mean / trials from logs).

Build + push the datagen/eval TPU image (`ghcr.io/open-thoughts/openthoughts-agent:tpu`) — the Iris TPU runtime (vLLM-TPU 0.20.0 PyTorch/XLA+JAX + Harbor + the Daytona sandbox backend) — AS AN IRIS KANIKO JOB on CoreWeave `cw-us-east-02a` (a CPU-only amd64 build; NO CUDA/nvcc, so it's minutes not hours, and the Mac CANNOT build it — arm64 + linux/amd64-only). Covers WHY kaniko not buildkit (shared with the gpu-rl skill), the crane-export-over-ubuntu recipe, and — the load-bearing lesson this skill exists for — how to make a canonical fast-moving dep (Harbor, our `marin-community/harbor` fork on `penfever/working`) ACTUALLY reach the worker: the RUNTIME gate is OT-Agent's `uv.lock` (the iris worker `uv sync --frozen --reinstall`s from it, OVERWRITING the image bake) — so deploying a harbor change is a `uv lock --upgrade-package harbor` + commit, NOT (only) an image rebuild; the Dockerfile `HARBOR_COMMIT` + `--force-reinstall` pin is a secondary image-consistency measure. VERIFY by grepping the INSTALLED file (

Condense and clarify an operational/research doc (SKILL.md, ops.md, tracker, README, agent_log) OR the COMMENTS of a launch/config YAML (.yaml/.yml) by an editor-subagent dispatched with fresh context and a list of file paths to review. The editor MOVES information to a concept-ordered taxonomy with functional subsections, then cuts paragraph-by-paragraph: stale gotchas + their corrections, deprecated features, rationalizations/justifications for actions (docs say WHAT to do, not WHY), partially-redundant sections, and flowery language. For YAML files: edit ONLY the `#` comment lines — every key, value, flag, list element, and the structural whitespace must stay byte-identical (the config is re-parsed as-is at launch). CONDENSE AND CLARIFY ONLY — never add or elaborate. Backs up every doc to ~/Documents/slop_docs/ before editing (dated + original filename), then leans toward OVER-condensing (backups exist). Returns a per-doc percent-shortened count + a brief overview of what was compressed for the supervisor

Preserve + publish a finished STANDARD (non-agentic GRPO) SkyRL RL checkpoint — the Delphi/rlvr/dapo math-and-reasoning cells launched via rl-standard-launch-leonardo (raw sbatch of hpc/skyrl_standard/leonardo/*, logger=console, NO Harbor/Daytona/trace_jobs). Covers: cancel pending retries, pick the BEST checkpoint by the trailing-5 EMA of reward via `parse_skyrl_metrics.py --format standard` (chain-aware, capped at the latest saved step), flatten weights to repo root, secret-scan, `hf upload` (Leonardo sbatch-tunnel) to laion/<run>-<step>-<size>B with the size suffix DERIVED FROM THE EXPORTED WEIGHTS (never the base-model name), DB register (--training-type RL + cross-user FK pre-check) ONLY for DB-registerable cells, clean up, and fire the Delphi downstream eval suite on the post-RL ckpt (defers to eval-standard-launch §5b). The ONLY artifacts are the model + the metric CSVs/report + the tracker scores — there is NO trace dataset. Use when a standard/non-agentic GRPO RL run finishes and needs uploading + (m

Launch, relaunch, or sweep STANDARD (non-agentic) SkyRL RL on CINECA Leonardo — GRPO on math/reasoning datasets (gsm8k, MATH/aime) and on-policy distillation (OPD, teacher→student) — via raw `sbatch` of the `hpc/skyrl_standard/leonardo/*` run scripts inside the writable apptainer SANDBOX + uv `marin_venv` (NOT `python -m hpc.launch`, NOT a `.sif`, NOT `--rl_use_conda`). Use when asked to run/relaunch a gsm8k or OPD GRPO canary, throughput/accuracy grid, or multi-node RL on Leonardo A100-64GB. Covers the GRPO/OPD knobs, the grid cell structure, the 1-node-vs-multi-node layout, the A100-64GB ceilings, and the no-internet/offline + gcc/HOME/Ray-temp-dir gotchas. For agentic Harbor+Daytona RL, this is the WRONG skill (Daytona needs internet — infeasible on Leonardo).

Launch SFT via `python -m hpc.launch --job_type sft` on any cluster (JSC Jupiter GH200, CINECA Leonardo A100, TACC Vista GH200), with EITHER backend — LLaMA-Factory (default) or axolotl (`--sft_backend axolotl`) — including Delphi tool-calling models (delphi template, tokenizer prep, jinja-as-ground-truth masking). This skill is the cluster-AGNOSTIC core (backend choice, Delphi handling, config maps, node-scaling, dataset mixing, cleanup recognition, common traps). Per-cluster particulars (preamble, paths, QOS/wall, sbatch patches, no-internet handling, HF-upload mechanics) live in `.claude/ops/<cluster>/ops.md §SFT`. Use when asked to SFT / launch a finetune / train a model on Jupiter, Leonardo, or TACC. Reference: notes/ot-agent/sft_experiments.md, CLAUDE.md.

Analyze the token length of an OT-Agent conversation-format (ShareGPT-style) dataset — the per-trace distribution (median/p90/max) and/or counts under a token threshold + a metadata predicate (e.g. "task_complete AND < 32768 tokens"). Use when asked how long traces are, how many fit a context window (32k/131k), or to filter a trace dataset by length + a field. Uses the OT-Agent analysis tools + the Qwen3-8B tokenizer. Runs LOCALLY on the Mac (no GPU); full-dataset tokenization of ~10k multi-turn traces takes a few minutes → run it in the background.

Run the full RL behavioral-analysis pipeline (scripts/analysis/analyze_rl_behavior.py) on a trained RL model to understand WHAT changed vs its pre-RL baseline, WHY, whether it PERSISTS, and its EVAL impact. Use when asked to "analyze RL behavior", "compare pre/post RL", "what did RL change", or to produce the Q1–Q4 behavioral report + GPT-5 judge for an `laion/...` (or any) RL checkpoint. Runs LOCALLY on the Mac (no GPU).

Detailed health check for a Levanter/executor TRAINING run on the marin Iris cluster (e.g. the delphi midtraining runs) — step progress vs target, loss/throughput, preemption + MAJOR step-gap detection, and checkpoint cadence. Use for an executor coordinator (`<run>-coord`) plus its nested `<run>-coord/checkpoints-<step>-<hash>` training child, which the harbor analyzer (analyze-job-history-iris) does NOT cover (training has no harbor trial sidecars, same as GPU-RL). Reads W&B per-step history + `iris job summary` + GCS checkpoints instead of harbor GCS output.

Build the gpu-rl Docker image (`ghcr.io/open-thoughts/openthoughts-agent:gpu-rl`) — the RL runtime for CoreWeave H100 (torch 2.11 + the from-source vLLM fork + flash-attn 2.8.3 + MarinSkyRL/torchtitan EP) — IN the CoreWeave `cw-us-east-02a` cluster, AS AN IRIS JOB USING KANIKO. Covers WHY kaniko not buildkit (the cluster denies CAP_SYS_ADMIN/bind-mounts + gVisor), the exact crane-export-over-ubuntu recipe, the load-bearing resource/flag settings (512GB node, `--single-snapshot`, `--cache`, ghcr GitHub-PAT creds), the Dockerfile gotchas baked into the build (uv `--index-strategy unsafe-best-match`, `python -m pip wheel`, torchtitan needs `tyro`), the build-asserts-as-validation, capturing the digest + bumping `DEFAULT_RL_DOCKER_IMAGE`, monitoring the build job, and WHEN a rebuild is actually required vs a runtime `--skyrl-ref` checkout. Use when asked to build / rebuild / push the gpu-rl image, after bumping the vLLM-fork commit / flash-attn / torch-CUDA / a baked dep. The Mac CANNOT build it (arm64 + needs li

EXECUTE a staged codebase plan (from code-create-staged-plan or an existing notes/<codebase>/ plan) one stage at a time, gate-by-gate, while keeping the local clone ground truth and a dated agent_logs/ progress log. For each stage: re-read the scope + reconfirm the (drifting) code anchors, make the edit in the LOCAL clone on the feature branch, run the validation gate (flag-off byte-identical FIRST, then behavior-on / parity / GPU smoke on the right SIF/env), commit+push and sync to the cluster (vLLM = build from source, never rsync), update the plan status, and only THEN advance. Log every stage/debug session in agent_logs/YYYY-MM-DD_<topic>.md so long runs don't lose context. Use when the user says "execute/run the plan", "do stage N", or "continue the <X> port/fix".

Read, aggregate, and (carefully) write OT-Agent eval/model data in the Supabase registry. Use when asked to look up a model's ID/OOD benchmark scores, build/refresh an ablation or paper table from eval results, find unevaluated models, register a model/eval, or reconcile duplicate rows. Covers the sandbox_jobs/models/benchmarks schema, the metrics-field shape gotchas, the ID/OOD benchmark master list + per-benchmark task counts (for binomial SE), and the multiple-entries-per-model rule.

Create a FRESH, snapshot-safe Harbor task dataset from an arbitrary input — a raw non-Harbor HF dataset (e.g. allenai/TMax-15K), a generator codebase (e.g. a GitHub repo like FrontierSmith), or just natural-language instructions with no seed data. The pipeline is a sequence of IDEMPOTENT stages — triage (pick the entry stage) → task-generation → Harbor conversion → snapshot-safe patcher → oracle-solution generation → quality gate — each with its OWN empirical test before advancing, intermediate artifacts uploaded to HF (laion/, parquet). The end artifact is a Harbor task dataset whose oracle (gold) solutions verify at a high rate, with ≤ 6 (hard ≤ 10) unique Daytona snapshots. Use when asked to "make/convert/build a task dataset", "turn <HF dataset> into Harbor tasks", "run <generator repo> into a task set", or "create tasks from these instructions". Runs LOCALLY on the Mac + Daytona (no GPU; teachers via API/vLLM). Distinct from datagen-reduce-dataset-snapshots (that fixes an EXISTING Harbor dataset's snapsh

Reduce the Daytona snapshot (unique-environment) count of a Harbor task dataset below the cap by editing its patcher's environment-build logic, without breaking task quality. Use when a dataset is flagged "SnapshotCapExceeded" / "N unique environments" with N over the threshold (target < 10), e.g. swegym at 906. The loop: set snapshot+oracle thresholds → count → diagnose the env-hash driver → group/unionize Dockerfiles in the patcher → regenerate + upload → re-count → TWO-TIER quality gate (harbor infra smoke + `--stages oracle` gold-patch yield) → navigate the snapshot↔fidelity tradeoff within a bounded iteration budget → record. Runs LOCALLY on the Mac + Daytona (no GPU).

Launch agentic Harbor evals through the OT-Agent unified eval listener (eval/unified_eval_listener.py) on any cluster: select models (query_unevaled_models.py / priority lists), wire the pinggy served-model tunnel, submit with the right preset + flags in tmux, then VERIFY the launch actually works via the 15-min infra sanity check (pinggy auth, Daytona→cluster api_base, vLLM POSTs, trial progression — catches "RUNNING but silently dead" jobs). Cluster-AGNOSTIC: per-cluster particulars (sbatch script, gpu-mem ceiling, concurrency, cert/tunnel, conda env, paths, Daytona key, pre-download) live in `.claude/ops/<cluster>/`. Use when asked to launch/relaunch agentic evals, or eval a model on a benchmark (terminal_bench_2 / dev_set_v2 / swebench / bfcl / aider).

Stand up a PUBLIC, shareable inference endpoint for an HF/gs model on an Iris TPU so people on the internet can vibe-test it in a browser or via an OpenAI-compatible API. Use when asked to "serve a model for testing", "throw up a public endpoint", "let people play with model X", or to demo a checkpoint. Combines marin-serve (marin#6556) + a Pinggy tunnel from our endpoint bank.

Clean up a completed NON-AGENTIC / HF-only SFT model — HF upload WITHOUT Supabase DB registration. The counterpart to sft-job-cleanup (which uploads AND registers). Use when a completed SFT cell belongs to an HF-only series (config `enable_db_registration: false`, or a known HF-only series like the Delphi #6279 54-grid): upload the weights to laion/ and STOP — do NOT run manual_db_push (DB-registering scaling-laws / sweep checkpoints pollutes the model registry; the artifacts are consumed by an eval grid, not the registry). Covers cell→hub_model_id resolution (incl. the Delphi launch_54_map.tsv), the Leonardo sbatch-tunnel upload, and the downstream eval hand-off. Use on discovering a completed Delphi/HF-only SFT cell during a sweep; for DB-registered models use sft-job-cleanup instead.

Delete stale Daytona sandboxes across ALL THREE orgs (DataComp, DataCompData, DataCompRL) in one pass. Runs `scripts/daytona/cleanup_stale_sandboxes.py` three times — once per org — each time setting the org's API key via `--api-key-env` so the script queries and deletes in the correct org. Default threshold is 60 minutes of inactivity; `--delete` actually removes them (dry run otherwise). Use when the Daytona sandbox count is climbing (eval/datagen/RL sandboxes left behind after jobs finish), when a sweep flags stale sandboxes, or when approaching the org snapshot/sandbox cap.

Launch the fixed Delphi #6279 RL-scaling-laws downstream MATH eval suite (MATH-500 / AIME24 / gsm8k via evalchemy + lm_eval) on CINECA Leonardo, for completed SFT / RL / base checkpoints. Covers finding which cells are newly-completed-but-uneval'd, the offline pre-download, the delphi_eval.sbatch invocation + RUN_NAME/STAGE convention, the load-bearing gotchas (chat-template override, 4k context, TP per head-count, MATH500/gsm8k split), and the SCORES.md tracker update. HF-upload-only — NEVER DB-register. Use when asked to eval Delphi #6279 SFT cells / update the scaling-laws score grid. Refs: experiments/active/delphi/rl-scaling-laws-6279/.

DESIGN a non-trivial codebase change (Harbor / MarinSkyRL / vLLM / OT-Agent / LLaMA-Factory) as a dependency-ordered STAGED PLAN before writing code — a feature port, a multi-step fix with parity requirements, a refactor, a kernel/perf change. Produces a parent plan doc + per-stage scope docs under notes/<codebase>/ (each stage = scope + GO/NO-GO validation gate + cost), with global invariants (flag-off byte-identical, parity gates), a borrow-map of code anchors (which drift), and safety considerations. Evidence/scoping breadcrumbs go in dated agent_logs/. Use when the user says "scope/plan this change", "stage it out", "design before coding", or a change is too big/risky for one shot. Pairs with code-execute-staged-plan (which runs the plan).

Publish + clean up a finished LLaMA-Factory SFT job on a no-internet HPC cluster (Jupiter/Leonardo): cancel pending retries, drop intermediate checkpoints, HF-upload the model to its configured --hub_model_id, register in Supabase via manual_db_push (--training-type SFT default), and free disk. Covers the 8B path (root safetensors, direct upload), the 32B/ZeRO-3 path (consolidate shards → safetensors first), the Qwen3.5 preprocessor_config copy, the don't-upload-partials policy, and the hf-upload gotchas (tmux not nohup, `hf upload` not `-large-folder`, Leonardo sbatch-tunnel not login node). Use when an SFT fine-tune finishes and needs uploading + registering, or "run the SFT cleanup checklist". Distinct from RL cleanup (rl-agentic-job-cleanup) and datagen cleanup (datagen-job-cleanup).