OpenThoughts-Agent

Post-run cleanup for a datagen (trace-generation) job on Iris/CoreWeave or an HPC cluster (Jupiter/Leonardo/Perlmutter): get the generated traces onto HF (penfever org) and free temporary disk. There is NO model checkpoint — the artifact is the trace dataset. Covers the TIMEOUT-strands-traces gotcha (uploads silently never run), the ONE-level trace_jobs nesting (vs RL's double-nest), the real-vs-failed sanity check (avg_turns ≈ 1.0 = dead run, don't upload), the otagent-env uploader, the non-empty HF verify, and safe disk cleanup (one-off task dirs vs shared canonical tasks; leave Daytona snapshots). Use when a datagen/trace job finishes (COMPLETED or TIMEOUT) and its traces need uploading + verifying, or when consolidating a chunked datagen run. Distinct from RL/SFT cleanup (those publish a model checkpoint).

Format HPC job-status reports as box-drawing tables, bucketed by job type (RL · SFT · Datagen · Eval · Catch-all), with the right metric columns, signal thresholds, and red-flags per bucket. Use whenever reporting active/recently-terminated job status — during a cron sweep, an ad-hoc "how are my jobs doing", or a single-job progress update. Covers which metrics are mandatory (entropy + collapse signals for RL, not just step/reward/grad), where to pull live status (SFT .out vs trainer_log.jsonl), the RL collapse-warning rule, and which log lines are benign noise vs real faults. Cluster-agnostic — resolve paths, active clusters, and credentials from .agents/ops at execution time.

Launch, monitor, and manually clean up a trajectory-generation (datagen) job on Marin's Iris TPU cluster via the OpenThoughts-Agent entrypoint. Use when asked to start, watch, rescue, or kill a datagen/tracegen run on Iris.

Preserve + publish a finished RL (SkyRL/GRPO) training checkpoint after the job terminates (completed at max_steps OR early-stopped/scancelled) on an HPC cluster (Jupiter/Leonardo/Perlmutter). Covers: cancel pending retries, pick the BEST checkpoint by trailing-5 EMA of reward across the full restart chain, flatten weights to repo root, secret-scan, `hf upload` to laion/<job>-<step>-<size>, Supabase DB register (--training-type RL + cross-user FK safety pre-check), upload training traces to penfever/<job>, parse metrics, and clean up. Use when an RL run needs its model uploaded + registered, or when asked to "run the RL cleanup checklist". Distinct from SFT cleanup (that's a different flow).

Build a clean per-dataset summary table/CSV for a datagen (trajectory-generation) campaign — one row per task source with Status (COMPLETED / FAILED / RUNNING / NOT STARTED), N Trials Completed, Mean Turns/Trace, Mean Tok/Trace, Mean Reward, and the HF trace-repo link. Use when asked to "summarize the campaign", "which datasets did we complete + their rewards/trials", "build a completion table/CSV", or to reconcile a prose tracker into auditable per-dataset metrics. Computes metrics by STREAMING each uploaded HF trace dataset (disk-bounded) and reusing the canonical OT-Agent analysis tools (scripts/analysis/utils.py: extract_conversation_text / count_turns / extract_reward) + the Qwen3-8B tokenizer; HF-ground-truths Status by probing each repo. Related skills: analyze-dataset-token-length (token method), analyze-job-history-iris (harbor Mean / trials from logs).

Analyze the token length of an OT-Agent conversation-format (ShareGPT-style) dataset — the per-trace distribution (median/p90/max) and/or counts under a token threshold + a metadata predicate (e.g. "task_complete AND < 32768 tokens"). Use when asked how long traces are, how many fit a context window (32k/131k), or to filter a trace dataset by length + a field. Uses the OT-Agent analysis tools + the Qwen3-8B tokenizer. Runs LOCALLY on the Mac (no GPU); full-dataset tokenization of ~10k multi-turn traces takes a few minutes → run it in the background.

Given a list of models (HF name stubs) that have valid agentic ID eval scores in Supabase, build a ranking table: raw per-benchmark accuracy on the 3 ID benchmarks (SWE-Bench-100, OT-TBLite=dev_set_v2, Terminal-Bench-2.0=tb2), HF links to each eval's trace dataset, and a NORMALIZED column = average per-benchmark z-score, ranked. Normalization matches the OpenThoughts-Agent paper (otagent-paper/02_arXiv/otagent.tex §Pipeline): per-benchmark z over the candidate set, averaged. Read-only. Use when asked to rank models / ablation arms by their ID evals the way the paper does.

Run the Iris harbor job-history analyzer (scripts/iris/analyze_iris_harbor_job.py) on a datagen/eval job and read its JSON sidecar for trustworthy throughput / preemption / productive-trial stats. Use whenever a status check needs REAL metrics (gen tok/s, cycles, non_empty rate, harbor exceptions) instead of an eyeballed log tail. It now queries the finelog log store directly (live ∪ GCS, deduped) — FAST (seconds, not minutes) and it ASSERTS completeness across all preempted attempts/generations, failing loud rather than returning fragments.

Run the full RL behavioral-analysis pipeline (scripts/analysis/analyze_rl_behavior.py) on a trained RL model to understand WHAT changed vs its pre-RL baseline, WHY, whether it PERSISTS, and its EVAL impact. Use when asked to "analyze RL behavior", "compare pre/post RL", "what did RL change", or to produce the Q1–Q4 behavioral report + GPT-5 judge for an `laion/...` (or any) RL checkpoint. Runs LOCALLY on the Mac (no GPU).

Detailed health check for a Levanter/executor TRAINING run on the marin Iris cluster (e.g. the delphi midtraining runs) — step progress vs target, loss/throughput, preemption + MAJOR step-gap detection, and checkpoint cadence. Use for an executor coordinator (`<run>-coord`) plus its nested `<run>-coord/checkpoints-<step>-<hash>` training child, which the harbor analyzer (analyze-job-history-iris) does NOT cover (training has no harbor trial sidecars, same as GPU-RL). Reads W&B per-step history + `iris job summary` + GCS checkpoints instead of harbor GCS output.

Build + push the datagen/eval TPU image (`ghcr.io/open-thoughts/openthoughts-agent:tpu`) — the Iris TPU runtime (vLLM-TPU 0.20.0 PyTorch/XLA+JAX + Harbor + the Daytona sandbox backend) — AS AN IRIS KANIKO JOB on CoreWeave `cw-us-east-02a` (a CPU-only amd64 build; NO CUDA/nvcc, so it's minutes not hours, and the Mac CANNOT build it — arm64 + linux/amd64-only). Covers WHY kaniko not buildkit (shared with the gpu-rl skill), the crane-export-over-ubuntu recipe, and — the load-bearing lesson this skill exists for — how to make a canonical fast-moving dep (Harbor, our `marin-community/harbor` fork on `main`) ACTUALLY reach the worker: the RUNTIME gate is OT-Agent's `uv.lock` (the iris worker `uv sync --frozen --reinstall`s from it, OVERWRITING the image bake) — so deploying a harbor change is a `uv lock --upgrade-package harbor` + commit, NOT (only) an image rebuild; the Dockerfile `HARBOR_COMMIT` + `--force-reinstall` pin is a secondary image-consistency measure. VERIFY by grepping the INSTALLED file (never trust

DESIGN a non-trivial codebase change (Harbor / MarinSkyRL / vLLM / OT-Agent / LLaMA-Factory) as a dependency-ordered STAGED PLAN before writing code — a feature port, a multi-step fix with parity requirements, a refactor, a kernel/perf change. Produces a parent plan doc + per-stage scope docs under notes/<codebase>/ (each stage = scope + GO/NO-GO validation gate + cost), with global invariants (flag-off byte-identical, parity gates), a borrow-map of code anchors (which drift), and safety considerations. Evidence/scoping breadcrumbs go in dated agent_logs/. Use when the user says "scope/plan this change", "stage it out", "design before coding", or a change is too big/risky for one shot. Pairs with code-execute-staged-plan (which runs the plan).

EXECUTE a staged codebase plan (from code-create-staged-plan or an existing notes/<codebase>/ plan) one stage at a time, gate-by-gate, while keeping the local clone ground truth and a dated agent_logs/ progress log. For each stage: re-read the scope + reconfirm the (drifting) code anchors, make the edit in the LOCAL clone on the feature branch, run the validation gate (flag-off byte-identical FIRST, then behavior-on / parity / GPU smoke on the right SIF/env), commit+push and sync to the cluster (vLLM = build from source, never rsync), update the plan status, and only THEN advance. Log every stage/debug session in agent_logs/YYYY-MM-DD_<topic>.md so long runs don't lose context. Use when the user says "execute/run the plan", "do stage N", or "continue the <X> port/fix".

Lint, run the pre-PR checks, commit, push, and author or update the branch's pull request in the required plain-text format. Use when committing, pushing, or creating/updating a PR.

Durably ARCHIVE everything informative from a finished run / experiment before it's cleaned up or its cluster artifacts age out — ALL Harbor trace_jobs (raw per-trial traces), ALL ray logs, ALL stdout/stderr (incl. vLLM/serving logs), and wandb. Pack-rat by design: if it's potentially informative, keep it. Only skip the non-informative-or-huge (model weights/checkpoints, core/memory dumps, massive raw tmux-pane / terminal-recording bytes). Many-tiny-files → tar THEN rsync (never rsync thousands of small files raw). Use when concluding/archiving an experiment, before a cleanup skill `rm`s an on-disk tree, or before CoreWeave R2/pod artifacts get GC'd. Per-run-type component maps live below; WHERE each artifact lives per cluster is a pointer into `.agents/ops/<cluster>/` and `.agents/projects/{harbor,marinskyrl,ot-agent}/`.

Read, aggregate, and (carefully) write OT-Agent eval/model data in the Supabase registry. Use when asked to look up a model's ID/OOD benchmark scores, build/refresh an ablation or paper table from eval results, find unevaluated models, register a model/eval, or reconcile duplicate rows. Covers the sandbox_jobs/models/benchmarks schema, the metrics-field shape gotchas, the ID/OOD benchmark master list + per-benchmark task counts (for binomial SE), and the multiple-entries-per-model rule.

Guardrailed DELETE of auto-registered eval `sandbox_jobs` rows that DID score but FAILED the harvest gate — partial evals (valid-complete <90% or non-benign infra-error >10%). These are the "DE-REGISTER candidate" rows the flawed_summ harvest defers. DISTINCT from `crud-purge-stale-eval-placeholders` (that removes NEVER-populated Pending/Started rows; this removes rows that populated stats/metrics but are below-gate). Use when asked to remove "partial / below-gate / <90%-completed-without-errors" evals owned by us. The MANDATORY parts: the authoritative gate utility (`scripts/database/eval_guardrail.py` — NEVER hand-roll the BENIGN set / error count), the cross-user FK-safety pre-check, the grandchild→child→job cascade, and REPORTING BACK the exact purged rows (§5) so the supervisor knows which state docs (e.g. flawed_summ STATE.md) to reconcile.

Safely purge stale, never-populated `sandbox_jobs` placeholder rows (eval launches that died/stalled before scoring) from the OT-Agent Supabase registry. Removes ONLY dead `Pending`/`Started` rows WE OWN that are >36h old with null `metrics`/`stats`/`ended_at`, via the mandatory cross-user FK-safety pre-check + the REQUIRED grandchild→child→job cascade delete (`sandbox_trial_model_usage` → `sandbox_trials` → `sandbox_jobs`). Other users' stale rows are REPORTED, never deleted. DRY-RUN first, then delete, then re-read. Use when the registry is clogged with dead placeholder eval rows, when a sweep flags stale Pending/Started/"Running" eval entries, or when the eval listener's dedup is mis-firing on dead rows. The general CRUD/read/aggregation skill is `crud-otagent-supabase`.

Create a FRESH, snapshot-safe Harbor task dataset from an arbitrary input — a raw non-Harbor HF dataset (e.g. allenai/TMax-15K), a generator codebase (e.g. a GitHub repo like FrontierSmith), or just natural-language instructions with no seed data. The pipeline is a sequence of IDEMPOTENT stages — triage (pick the entry stage) → task-generation → Harbor conversion → snapshot-safe patcher → oracle-solution generation → quality gate — each with its OWN empirical test before advancing, intermediate artifacts uploaded to HF (laion/, parquet). The end artifact is a Harbor task dataset whose oracle (gold) solutions verify at a high rate, with ≤ 6 (hard ≤ 10) unique Daytona snapshots. Use when asked to "make/convert/build a task dataset", "turn <HF dataset> into Harbor tasks", "run <generator repo> into a task set", or "create tasks from these instructions". Runs LOCALLY on the Mac + Daytona (no GPU; teachers via API/vLLM). Distinct from datagen-reduce-dataset-snapshots (that fixes an EXISTING Harbor dataset's snapsh

Launch a datagen (trace-generation) job on an HPC cluster (Jupiter/Leonardo/Perlmutter) via the OpenThoughts-Agent `hpc.launch --job_type datagen` entrypoint — the cluster-AGNOSTIC general flow: extract tasks from a parquet, then submit a managed vLLM-serve + Harbor/Daytona trace run that uploads the trajectories to an HF repo. Use when asked to start a datagen / trace-generation job, generate agent traces from a task dataset, or advance / start a row in the MiniMax-M2.7 datagen tracker. For the Iris TPU path use **datagen-launch-iris** instead; per-cluster particulars (ssh, paths, conda env, which vllm-serve config matches the cluster's GPUs, JSC pre-download) live in `.agents/ops/<cluster>/`.

Reduce the Daytona snapshot (unique-environment) count of a Harbor task dataset below the cap by editing its patcher's environment-build logic, without breaking task quality. Use when a dataset is flagged "SnapshotCapExceeded" / "N unique environments" with N over the threshold (target < 10), e.g. swegym at 906. The loop: set snapshot+oracle thresholds → count → diagnose the env-hash driver → group/unionize Dockerfiles in the patcher → regenerate + upload → re-count → TWO-TIER quality gate (harbor infra smoke + `--stages oracle` gold-patch yield) → navigate the snapshot↔fidelity tradeoff within a bounded iteration budget → record. Runs LOCALLY on the Mac + Daytona (no GPU).

Launch NON-AGENTIC (standard) data generation — plain vLLM/API completion generation with NO Harbor agent loop or Daytona sandboxes. Two paths: (1) Curator sharded datagen — the multi-node data-parallel run_curator_datagen_sharded.sbatch (one vLLM server per node, disjoint dataset slices, auto-resume, manual afterany restart chain, `curator` conda env, --account=reformo); (2) the declarative `generate.py` / class-based `generate_abstract.py` generators under data/ (data/generation BaseDataGenerator + InferenceEngine for OpenAI/Anthropic/vLLM). Use for bulk completion/synthetic-data generation. The AGENTIC trace-generation path (Harbor + Daytona rollouts, MiniMax/GLM trace sets) is the SEPARATE `datagen-launch` skill.

Debug a code bug with a structured debug log that records hypotheses, changes, and results.

Condense and clarify an operational/research doc (SKILL.md, ops.md, tracker, README, agent_log) OR the COMMENTS of a launch/config YAML (.yaml/.yml) by an editor-subagent dispatched with fresh context and a list of file paths to review. The editor MOVES information to a concept-ordered taxonomy with functional subsections, then cuts paragraph-by-paragraph: stale gotchas + their corrections, deprecated features, rationalizations/justifications for actions (docs say WHAT to do, not WHY), partially-redundant sections, and flowery language. For YAML files: edit ONLY the `#` comment lines — every key, value, flag, list element, and the structural whitespace must stay byte-identical (the config is re-parsed as-is at launch). CONDENSE AND CLARIFY ONLY — never add or elaborate. Backs up every doc to ~/Documents/slop_docs/ before editing (dated + original filename), then leans toward OVER-condensing (backups exist). Returns a per-doc percent-shortened count + a brief overview of what was compressed for the supervisor

Audit + recover a finished agentic eval. ALWAYS start with the read-only, idempotent completeness/health audit (§0): job finished? score present + non-zero + not obviously broken? HF traces present + linked? trial count ≈ n_rep × benchmark_size? — it writes nothing and recommends an action per check. Then run only the flagged remediations: manual HF-trace upload + Supabase DB-registration via manual_db_eval_push.py, the vLLM-numeric-ID → real-HF-model-name fix (with cross-user FK safety), verify, free disk. Use to verify an eval is truly complete, when a sweep finds an eval that didn't upload/register or has a broken/zero score or short trial count, or to re-register/correct an eval's model/traces. Distinct from the model- publishing cleanups (rl-agentic-job-cleanup / sft-job-cleanup) and datagen-job-cleanup — this is the EVAL path.

Launch, monitor, and manually clean up an eval job on Marin's Iris TPU or CoreWeave H100x8 GPU cluster via the OpenThoughts-Agent entrypoint. Use when asked to start, watch, or kill a model evaluation (evalchemy / agent-harness benchmarks) on Iris.

Launch agentic Harbor evals through the OT-Agent unified eval listener (eval/unified_eval_listener.py) on any cluster: select models (query_unevaled_models.py / priority lists), wire the pinggy served-model tunnel, submit with the right preset + flags in tmux, then VERIFY the launch actually works via the 15-min infra sanity check (pinggy auth, Daytona→cluster api_base, vLLM POSTs, trial progression — catches "RUNNING but silently dead" jobs). Cluster-AGNOSTIC: per-cluster particulars (sbatch script, gpu-mem ceiling, concurrency, cert/tunnel, conda env, paths, Daytona key, pre-download) live in `.agents/ops/<cluster>/`. Use when asked to launch/relaunch agentic evals, or eval a model on a benchmark (terminal_bench_2 / dev_set_v2 / swebench / bfcl / aider).

Consolidate FINISHED standard / lm_eval (evalchemy) math-suite eval jobs — the Delphi #6279 MATH-500 / AIME24 / gsm8k grid launched via eval-standard-launch — into the SCORES.md tracker. Per job: confirm a NON-EMPTY seed42 result (a crash leaves it empty — don't record it as a score), rsync the results_*.json into the local per-model dir, write scalar partials, parse the scalars (MATH-500 acc×100 / AIME24 10-seed mean±sd / gsm8k strict+flex / Raw), and flip the SCORES.md row 🚀 eval submitted → ✅ done. The artifact is SCALAR SCORES IN A TRACKER — HF-upload-only, NEVER DB. Use when asked to consolidate / harvest finished standard math-eval jobs or fill the scaling-laws score grid. DISTINCT from eval-agentic-cleanup (the Harbor-trace + Supabase DB path).

Launch the fixed Delphi #6279 RL-scaling-laws downstream MATH eval suite (MATH-500 / AIME24 / gsm8k via evalchemy + lm_eval) on CINECA Leonardo, for completed SFT / RL / base checkpoints. Covers finding which cells are newly-completed-but-uneval'd, the offline pre-download, the delphi_eval.sbatch invocation + RUN_NAME/STAGE convention, the load-bearing gotchas (chat-template override, 4k context, TP per head-count, MATH500/gsm8k split), and the SCORES.md tracker update. HF-upload-only — NEVER DB-register. Use when asked to eval Delphi #6279 SFT cells / update the scaling-laws score grid. Refs: experiments/active/delphi/rl-scaling-laws-6279/.

File a GitHub issue for a bug or improvement found this session.

The PROCEDURE for one every-3-hours Iris job-status sweep — primarily the marin TPU datagen/eval jobs ("iris" here = the marin TPU cluster), plus CoreWeave GPU-RL as monitor-only. Query both clusters, run the harbor analyzer on each active TPU datagen/eval job, classify every job (datagen / eval / other / GPU-RL) and apply its treatment, then take the standing DATAGEN-ONLY autonomous actions (auto-rescue, keep-2-in-flight). This is the methodology the recurring cron prompt runs; the cron itself is (re)installed via monitor-restore-iris. Use for "run an iris sweep / cluster sweep now" or as the reference behind each cron tick.

Produce a comprehensive cross-cluster job-status update for a recurring N-hourly cluster sweep. Gather squeue/sacct on each cluster (validating against false-drain), bucket every active + recently-terminated job by type (RL / SFT / datagen / eval / catch-all), pull each type's signals, render them in the job_monitor_table.md formats, and flag completions (→ the matching cleanup skill), genuine failures (→ diagnose + agent_logs), and per-type health red-flags. Cluster-AGNOSTIC — ssh strings, code/log/exp paths, concurrency caps, gpu-mem ceilings live in `.agents/ops/<cluster>/`. Use for "run a cluster sweep", the N-hourly cron, or "give me a status update on all jobs".

Re-register the every-3-hours Iris job-monitor cron (status check + datagen auto-rescue/keep-2-in-flight) if it has been lost. Primarily the marin TPU datagen/eval jobs ("iris" = the marin TPU cluster); also queries CoreWeave (cw-us-east-02a) GPU-RL as monitor-only. The cron is session-only and recurring crons auto-expire after 7 days, so it's routinely lost on a session restart. Use at the start of a new session, after a restart, or when the user asks to restore/check the iris monitoring cron. The sweep PROCEDURE the cron runs lives in monitor-cron-sweep-iris.

Re-create the local 3-hour tri-cluster cluster-sweep loop (Leonardo + CoreWeave(iris) + TACC(Vista); Jupiter SKIPPED until ~Jul 12) — the autonomous ML-ops monitor — if it has been lost. The loop is session-only and is dropped on any session restart, so re-establish it at the start of a new session or whenever the user asks to restore/restart the 3h sweep/cron/monitor. Sets a /loop 3h (or equivalent recurring cron) whose task is the canonical sweep prompt below: status-table active/pending/completed jobs, auto-cleanup+DB-register completions, diagnose+remediate failures via subagents, log each launch/state-change as a standalone dated file to agent_logs/. The prompt block here is the source of truth — copy it verbatim.

Re-register the every-3-hours UNIFIED OPS TICK cron — the CURRENT operator-owned monitor for the qwen3.5-122b-131k-datagen-opencode campaign (keep-3 datagen with autonomous rescue+refill) AND the delphi midtrain 1e23_p33m67_k0p20 (monitor-only), plus a marin+CoreWeave sweep and standing Daytona snapshot cleanup. The cron is session-only and recurring crons auto-expire after 7 days, so it's routinely lost on a session restart. Use at the start of a new session, after a disconnect/restart, or when the user asks to restore the monitor cron. Supersedes monitor-restore-iris (which is the OLDER datagen-OUT-of-scope variant); use THIS skill when the current session owns the 131k datagen campaign + midtrain. The per-tick sweep methodology lives in monitor-cron-sweep-iris; the table format in monitor-job-tables.

Launch / relaunch agentic MarinSkyRL (SkyRL GRPO) RL on Marin's Iris / CoreWeave GPU cluster (cw-us-east-02a, 8x H100-80GB + InfiniBand per node) via `python -m cloud.iris.launch_rl_iris` (run from the MarinSkyRL repo root) + the gpu-rl Docker image (NO Apptainer SIF). Covers the MoE 30B-A3B arms (CP + DCP=2 + R3 @ 131k) and the dense 8B arms (seqnorm + TIS); the iris configs default to the **Megatron** training backend (`trainer.strategy: megatron`, FSDP2 selectable) — the exact launcher flag set (`--rl_config`, `--model_path`, `--train_data`, `--num-nodes`, `--rendezvous-dir`, `--job-name`, `--priority`, `--cpu`, `--max-retries`), the gang/leafgroup/Kueue multi-node Ray rendezvous, the iris config-authoring rules (NO container block, load-bearing top-level `extra_env:` forwarding, disaggregated placement + explicit `num_inference_engines`, SIF→Docker env translation), and the bring-up gotchas (`--cpu 48`, `--max-retries ≥1`). Use when asked to launch / relaunch an agentic SkyRL RL run on Iris / CoreWeave. C

Launch / relaunch agentic RL (SkyRL terminal_bench + Harbor + Daytona) on JSC Jupiter (GH200). Covers the dense 8B/32B FSDP2 arms (seqnorm, TIS, shaped, symclip, lrboost, loopshape) and the MoE/80B Megatron arms (Qwen3-Coder-30B-A3B, Qwen3-Next-80B-A3B) — the exact `python -m hpc.launch --job_type rl` flag set, which flags vary per arm (config / model_path / train_data / num_nodes), runtime+SIF selection, the Daytona RL-org + chain-restart conventions, and the standing constraints (≤6 RL/cluster, a3 CONCLUDED, TIMEOUT restarts are normal). Use when asked to launch / relaunch / refill an agentic SkyRL RL run on Jupiter. Reference: notes/ot-agent/rl_experiments.md, .agents/ops/jupiter/{ops.md,ENVIRONMENT_MAP.md}.

Deep single-RL-job health probe → a KILL / NO-KILL / ERROR recommendation for the supervisor. Dispatched as a subagent on every monitor tick for RL jobs in NEW/UNTESTED settings (new config/geometry/model, "debug" or "smoke-test" flavor, first launches after a code/config change), and whenever a running job looks starved or wedged. Goes BEYOND state-poll + table metrics: captures the job's logs + trace_jobs + GPU view, then runs four gates — (A) liveness, (B) resource utilization / engine subscription, (C) rollout quality — and emits ONE evidence-backed verdict. The subagent NEVER kills — it recommends; the supervisor owns the kill. This skill holds the METHODOLOGY only; every cluster-access and codebase fact is a POINTER into .agents/ops/<cluster>/ and .agents/projects/<dep>/ (those are the single source of truth — do not re-encode them here, they go stale).

Preserve + publish a finished STANDARD (non-agentic GRPO) SkyRL RL checkpoint — the Delphi/rlvr/dapo math-and-reasoning cells launched via rl-standard-launch-leonardo (raw sbatch of hpc/skyrl_standard/leonardo/*, logger=console, NO Harbor/Daytona/trace_jobs). Covers: cancel pending retries, pick the BEST checkpoint by the trailing-5 EMA of reward via `parse_skyrl_metrics.py --format standard` (chain-aware, capped at the latest saved step), flatten weights to repo root, secret-scan, `hf upload` (Leonardo sbatch-tunnel) to laion/<run>-<step>-<size>B with the size suffix DERIVED FROM THE EXPORTED WEIGHTS (never the base-model name), DB register (--training-type RL + cross-user FK pre-check) ONLY for DB-registerable cells, clean up, and fire the Delphi downstream eval suite on the post-RL ckpt (defers to eval-standard-launch §5b). The ONLY artifacts are the model + the metric CSVs/report + the tracker scores — there is NO trace dataset. Use when a standard/non-agentic GRPO RL run finishes and needs uploading + (m

Launch, relaunch, or sweep STANDARD (non-agentic) SkyRL RL on CINECA Leonardo — GRPO on math/reasoning datasets (gsm8k, MATH/aime) and on-policy distillation (OPD, teacher→student) — via raw `sbatch` of the `hpc/skyrl_standard/leonardo/*` run scripts inside the writable apptainer SANDBOX + uv `marin_venv` (NOT `python -m hpc.launch`, NOT a `.sif`, NOT `--rl_use_conda`). Use when asked to run/relaunch a gsm8k or OPD GRPO canary, throughput/accuracy grid, or multi-node RL on Leonardo A100-64GB. Covers the GRPO/OPD knobs, the grid cell structure, the 1-node-vs-multi-node layout, the A100-64GB ceilings, and the no-internet/offline + gcc/HOME/Ray-temp-dir gotchas. For agentic Harbor+Daytona RL, this is the WRONG skill (Daytona needs internet — infeasible on Leonardo).