| name | rl-job-health-deep-dive |
| description | Deep single-RL-job health probe → a KILL / NO-KILL recommendation for the supervisor. Dispatched as a subagent on every monitor tick for RL jobs in NEW/UNTESTED settings (new config/geometry/model, "debug" or "smoke-test" flavor, first launches after a code/config change). Goes BEYOND state-poll + the table metrics: syncs the job's trace_jobs + stderr/stdout + Ray logs to ~/Documents/experiments/traces via the EXISTING capture tool (CoreWeave: scripts/iris/peek_rl_rollouts.sh `pull`), then runs four gates — (1) liveness (tail stdout/stderr: zombie/wedged/dead?), (2) resource utilization (live-poll GPUs: all inference engines alive + generating at a hardware/model-size-reasonable cadence per the serving LUT; training stage not VRAM/RAM-OOM), (3) rollout quality (trace_jobs: trials init/complete, non-zero rewards, turns completing, agent outputs sane, tasks hard, verifiers firing), and emits ONE verdict with evidence + next steps. The subagent NEVER kills — it recommends; the supervisor owns the kill (standing guardrail). Cluster-agnostic; defers access/hardware/log-path particulars to .claude/ops/<cluster>/ and dependency facts to .claude/projects/<dep>/. Reference: scripts/iris/peek_rl_rollouts.sh, scripts/iris/watch_job_state.py, .claude/ops/iris/coreweave_gpu_ops.md, .claude/skills/rl-agentic-launch-iris §8. |
rl-job-health-deep-dive
A deep, single-RL-job health probe that ends in one recommendation to the supervisor: KILL or
NO-KILL, backed by hard evidence and next steps. This is the heavier, per-job complement to the breadth
sweep: monitor-cron-sweep looks at every job briefly; this skill looks at ONE RL job hard and is the
right tool when a run is in a new or untested setting (new config / geometry / model / image, a "debug" or
"smoke-test" launch, or the first launch after a code/config change) where state-poll + table metrics are
necessary but not sufficient to tell "genuinely progressing" from "silently dead."
You are a SUBAGENT producing a recommendation — you do NOT execute the kill. Standing guardrail
(supervisor-init, every launch skill): never kill a RUNNING job without explicit permission. Your
deliverable is the KILL/NO-KILL verdict + reasoning + recommended next steps. The supervisor decides and,
if KILL, runs iris job kill … / scancel … themselves. The ONLY exception is the supervisor's own standing
autonomy over our own deterministically-doomed / wedged jobs — and even then the decision is theirs, on
your evidence. When in doubt, recommend NO-KILL + escalate (a wrongly-killed healthy run wastes a whole
bring-up; a wrongly-kept dead one wastes one more sweep — asymmetric).
Resources you must use (don't re-derive)
.claude/ops/<cluster>/ — machine/cluster particulars: access (kubeconfig/ssh), log-path discovery,
GPU-poll mechanics, gpu-mem ceilings, the binding gotchas. CoreWeave → ops/iris/coreweave_gpu_ops.md
(+ coreweave_h100_cloud_hardware.md for the node shape); Leonardo → ops/leonardo/ops.md; TACC →
ops/tacc/ops.md. Read the relevant one FIRST — it tells you how to reach the job and read its logs
safely (GPFS find/du ban, login-node false-drain, the kubeconfig export, the otagent iris binary)..claude/projects/<dep>/ — what each codebase is + its facts/gotchas: marinskyrl/ (the SkyRL/GRPO
trainer — log-line vocabulary, sel_rows/EPDIAG, weight-sync), vllm/ (the serve engine — the fork's MoE/DCP/R3
flags, throughput knobs, enforce_eager), harbor/ (the rollout/trial layout + passthrough_exceptions),
daytona/ (sandbox/reward-0 failure modes). Use these to read the logs correctly, not by guesswork.scripts/iris/peek_rl_rollouts.sh — the EXISTING capture tool for CoreWeave (§1). Do NOT hand-roll an
ad-hoc R2/kubectl pull.scripts/iris/watch_job_state.py — the authoritative CoreWeave lifecycle state-poll (§2).rl-agentic-launch-iris §8 — the per-rung CoreWeave bring-up ladder this skill operationalizes; cite it
rather than restating every milestone string.
0. Inputs + setup (gather these first)
You need, from the dispatching supervisor (ask only if genuinely missing — most are derivable):
| Input | How to get it | Used for |
|---|
| Cluster | from the dispatch | which ops/<cluster> to read + which capture/poll path |
| Job id | from the dispatch (/benjaminfeuer/<job> on CoreWeave; SLURM jobid on Leonardo/TACC) | state-poll, log path |
| Pod-name substring (CoreWeave) | the --job-name slug (pod name = iris-benjaminfeuer-<slug>-<rank>-<hash>-<gen>) | peek_rl_rollouts.sh + kubectl exec |
| Model + size (B, dense vs MoE active-B) | from the config / --model_path | the serving-throughput LUT (§3) |
| Stage (bring-up / inference / training) | from the logs you capture in §1–2 | which §3 check applies |
| What's "new/untested" | from the dispatch (e.g. "TP=2+EP=2 first run", "R3+DCP unvalidated", "post-weight-sync-fix") | what to scrutinize hardest in §4 |
Environment (CoreWeave example — adapt per ops/<cluster>):
source "${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV to the secrets file first}"
export KUBECONFIG=~/.kube/coreweave-iris-gpu
PY=/Users/benjaminfeuer/miniconda3/envs/otagent/bin/python
IRIS=/Users/benjaminfeuer/miniconda3/envs/otagent/bin/iris
All iris/kubectl calls are SYNCHRONOUS — never background them (ops/iris/coreweave_gpu_ops.md).
Restart-burn check (do this EARLY — it sets the syncdown scope in §1 and feeds the verdict). Before anything
else, find out whether this job has already burned a restart/retry — because (a) a run that has already
failed once is a sharply different risk than a clean first attempt, (b) the prior-attempt failure is
high-signal for the verdict, and (c) the remaining retry budget bounds urgency (a run on its last retry
that's misbehaving is closer to a KILL than one with headroom). Cheap signals (no heavy capture yet):
$PY scripts/iris/watch_job_state.py /benjaminfeuer/<job> --once --json
$IRIS --cluster=cw-us-east-02a job summary /benjaminfeuer/<job> --json
kubectl get pods -n iris -o name | grep "<slug>" | sed -E 's/.*-([0-9]+)$/gen \1/' | sort -u
Compare burned-count to the launch's --max-retries K (we launch with K=1) → remaining = K − burned.
SLURM (Leonardo/TACC): the restarts are the afterany chain-restart legs — sacct -u <user> --name <jobname> -X --format=JobID,State,Start,End,ExitCode lists every chain link; each prior terminal (FAILED/TIMEOUT/CANCELLED)
leg before the running one is a burned attempt (a clean TIMEOUT→successor is a normal chain restart, NOT a burn —
distinguish: a wall-clock TIMEOUT with a healthy successor ≠ a crash-and-retry). scontrol show job <id> Restarts=
also counts requeues.
Record restarts burned = B / max K (remaining = K−B) and, if B>0, the terminal state + exit/error of each
prior attempt (from the summary/sacct). If B>0, set the syncdown in §1 to include the failed runs (next
section), and carry B, the remaining budget, and "same failure each attempt?" into Gate A (§2) and the verdict
(§5): the same crash repeating across every restart = deterministically doomed → KILL; a restart burned on a
genuine transient (e.g. an HF-weight-resolution flake the retry-wrapper now catches) that is now healthy = benign.
1. Sync the artifacts (use the existing tool — never ad-hoc)
Pull trace_jobs + stderr/stdout + (if available) the Ray logs into the canonical capture dir
~/Documents/experiments/traces/. Use the dependency's own capture tool — do not hand-roll a kubectl/R2
sync (the tool already handles per-object size-verify, the latest-generation pod, the REMOTE-R2-vs-node-local
trials_dir split, and a provenance MANIFEST).
CoreWeave:
IRIS_BIN=$IRIS bash scripts/iris/peek_rl_rollouts.sh <pod-name-substring> pull
The per-rank pod_rank*.log files are the engine/Ray stderr+stdout (k8s merges container stdout/stderr).
For deeper Ray actor logs, kubectl exec -n iris <pod> -c task -- bash -lc 'ls /tmp/ray/session_latest/logs'
and pull any worker-*.err / raylet.* of interest (the finelog usually surfaces the real traceback first).
If the §0 restart-burn check found B>0 burned restarts → the syncdown MUST include the FAILED runs, not
just the live generation. The default pull captures the current (latest-generation) pods' logs + the
shared trials — but the prior attempts are exactly the evidence you need. Capture them:
- iris finelog spans ALL generations — it is the durable record of every burned attempt's bring-up +
crash (the GC'd prior-generation pods' stdout is gone, but the finelog kept it).
peek … pull already
pulls it --no-tail full-history; confirm it covers the original submission (if it was tailed, re-pull
with $IRIS … job logs /benjaminfeuer/<job> --since-ms <ORIGINAL submitted_at_ms> --no-tail, using the
first submission time, not the latest generation's). Then grep it for each prior attempt's terminal
traceback — the real reason each restart was burned (and whether it's the SAME failure every time).
- The REMOTE R2 trials_dir is shared across generations (
s3://marin-us-east-02a/iris/<job>/trace_jobs —
the storage root resolves via marin_prefix(), see .claude/ops/iris/coreweave_gpu_ops.md §rendezvous; don't hardcode the region bucket), so
pull already grabbed failed-generation trials too — segregate/label them by episode/timestamp so the
§4 rollout read doesn't conflate a dead generation's reward-0 trials with the live one's.
- SLURM: the failed legs are separate job IDs in the
afterany chain — rsync the .out/.err of
each prior terminal leg (its own jobid, located via scontrol/sacct per §0) into the capture dir,
alongside the running leg's. Don't sync only the live leg.
SLURM (Leonardo/TACC): there is no peek tool — the equivalent is: locate the .out/.err via
scontrol show job <id> -o (StdOut=/StdErr=/%Z workdir — never find/du on GPFS, per
ops/<cluster>), then rsync/scp the .out, .err, and the run's trace_jobs/ tree into
~/Documents/experiments/traces/<job>_<stamp>/. (Most "new/untested" RL right now is CoreWeave; the four
gates below are cluster-agnostic once you have logs + trials + a GPU view.)
If pull returns 0 trials / no logs: that is itself a signal, not a tool failure — usually the rank-0
pod is gone (terminal job → node-local trials GC'd) or was just (re)started. Cross-check with §2's state-poll
before concluding; a fresh launch legitimately has 0 completed trials for a while (long episodes).
2. Gate A — Liveness (is it alive, or zombie/wedged/dead?)
Liveness = authoritative STATE-POLL + a log-freshness read — NEVER a single log-string grep (a clean
kill/eviction/preempt emits no terminal string and reaps pods, so a content-watch sits idle while the job is
gone — the exact miss rl-agentic-launch-iris §8 warns about).
$PY scripts/iris/watch_job_state.py /benjaminfeuer/<job> --once --json
Then read the captured logs (§1) for freshness + wedge signatures:
- Log freshness. Compare the newest meaningful log line's timestamp to "now" against the run's expected
cadence (a generating engine prints
Avg generation throughput … every few seconds; a training step lands
on the order of minutes). Materially-stale-beyond-cadence = suspected wedge (engine deadlock, NCCL
stall, generation-buffer starvation) — a job can hold its whole gang for hours while hung and still read
running, pods=N.
- Wedge / death signatures (grep the finelog +
pod_rank*.log): a real Traceback, CUDA out of memory
/ OutOfMemoryError, RayActorError / Raylet … died / worker SIGKILL/SIGABRT, an NCCL hang
(WorkNCCL(...Timeout(ms)=...) ran for N ms before timing out, ProcessGroupNCCL preparing to dump debug info), an RPC / sample_tokens / execute_model timeout, EngineDeadError.
- Benign noise — do NOT read as death (per
marinskyrl/vllm projects + monitor-job-tables):
shm_broadcast … No available shared memory broadcast block found in 60/600s (engine idle-waiting — heartbeat,
not a kill), a transient ghcr.io blob EOF → ImagePullBackOff self-heal, the Unknown vLLM environment variable VLLM_ALLOW_ROUTED_EXPERTS_DCP whitelist note, … 32769 input tokens > 32768 max /
ContextLengthExceededError / AgentTimeoutError (harbor passthrough — appear in healthy runs),
rollout_train_prob_diff_mean in the millions (outlier-dominated, normal). opCount dead is benign debug-token
noise, not EngineDeadError.
⚠️ THE COLOCATED-ENGINE DECEPTION — a wedged POLICY mesh that falsely reads ALIVE (learned 2026-06-28, BOTH
MoE-EP8 arms wedged ~1h apart, undetected for hours). In a disaggregated/colocated RL job the vLLM
inference engines + RolloutCoordinator are SEPARATE Ray actors from the policy FSDP mesh. When the
POLICY mesh hangs in a NCCL collective (ProcessGroupNCCL watchdog got stuck for 1800s → SIGABRT on rank-0 →
gang wedges), the engines KEEP GENERATING — so every cheap "liveness" signal LIES: (a) the state-poll
still reads running / pods=N / failure_count=0 (pods alive, just not computing); (b) the wandb heartbeat
stays FRESH (the engine/coordinator actors keep emitting system metrics — wandb liveness is NOT policy-mesh
liveness); (c) GPU util reads HIGH (28–96%) on the engine-colocated pods (engines busy-generating
rollouts no training step will ever consume). A util-only read MIS-diagnoses this as ALIVE-SLOW. The three
signals that DON'T lie — check ALL of them every probe on any multi-mesh (FSDP×EP×CP) RL run, proactively, do
NOT wait for util to look wrong:
- Ray logs (authoritative for actor/worker death). kubectl-exec rank-0 (
-c task):
grep -iE 'died|SYSTEM_ERROR|connection error code 2|End of file|SIGABRT|Aborted|raylet.*fail|ActorDied' /tmp/ray/session_latest/logs/{raylet.*,python-core-*,worker-*.err}. A worker died … SYSTEM_ERROR … connection error code 2 … End of file = the policy worker is DEAD (Ray may even respawn it and it re-hangs → a SECOND watchdog-stuck).
- The NCCL watchdog in the finelog —
ProcessGroupNCCL('s)? watchdog got stuck for <N> seconds / Fatal Python error: Aborted / WorkNCCL(SeqNum=…,OpType=…) ran for … before timing out / the TCPStore … sendBytes failed … Broken pipe cascade across FSDP ranks. The 1800s watchdog makes a hang detectable within ~30 min of onset — grepping this each probe is the FASTEST catch (we set TORCH_NCCL_DUMP_ON_TIMEOUT → a flight-recorder dump also lands; capture it before any kill).
- Is the TRAINER/DRIVER log ADVANCING (not just engine generation)?
mesh_fsdp / run_training / fwd_logprobs / the step counter must MOVE; a frozen trainer + spinning engines = wedge. Separate POLICY-mesh GPUs from the colocated ENGINE GPUs — a wedged policy reads 0% on the policy ranks even while engine GPUs show util (identify policy vs engine pods/GPUs from the placement; don't average them).
A running / fresh-heartbeat / high-util reading is necessary-but-NOT-sufficient for a multi-mesh RL job —
clear all three above before calling Gate A PASS. NB: this is a NCCL-collective hang, NOT an OOM (memory.peak
held ~35% when it bit) — rule OOM out, but never let a clean memory read imply liveness.
Gate A verdict:
- DEAD / TERMINAL (state-poll says failed/killed/absent, or 0 pods) → there's nothing to kill; report
TERMINAL + the root-cause traceback + whether it's transient (relaunch-worthy) or deterministic.
- WEDGED (RUNNING/pods present but a real hang signature + stale-beyond-cadence logs, no benign explanation)
→ lean KILL (it's burning a multi-node gang doing nothing). Carry the evidence to §5.
- ALIVE + fresh logs → proceed to Gate B.
3. Gate B — Resource utilization (are the GPUs actually working?)
Live-poll the GPUs (don't infer from logs alone). CoreWeave — exec nvidia-smi on every rank:
for p in $(kubectl get pods -n iris -o name | grep "<job>" | sed 's#pod/##'); do
echo "== $p =="
kubectl exec -n iris "$p" -c task -- nvidia-smi \
--query-gpu=index,utilization.gpu,memory.used,memory.total,power.draw --format=csv,noheader
done
(SLURM: srun --jobid=<id> --overlap -w <node> nvidia-smi …, or read the .out's periodic util/mem prints —
see ops/<cluster>.) Interpret by stage (which stage the §1–2 logs put you in):
B-inference (rollout/generation stage)
Every inference engine must be alive, fed, and generating at a cadence reasonable for the hardware + model
size. Concretely:
- All engines present + busy. Engine count =
num_inference_engines × TP GPUs should show non-trivial
utilization.gpu (decode is bursty, but a generating engine is not pinned at 0%). A GPU at 0% util with
memory resident across several polls = an engine that loaded weights but is not generating (starved /
wedged / never received requests).
- vLLM actually emitting tokens. In the logs:
Avg generation throughput: >0 tokens/s, Running: R reqs, Waiting: W recurring on each engine. This is the literal "is vLLM firing" check.
- Throughput is hardware-reasonable — compare the aggregate
tokens/s (summed across an engine's Running
reqs) to the serving LUT below. Below-floor throughput while Running > 0 is a red flag (eager mode,
bad geometry, oversubscription thrash, or a degraded weight-sync producing pathological outputs).
- Queue is draining, not thrashing.
Waiting ≈ 0 (or falling) = healthy. Waiting persistently ≫
Running with flat/low throughput = the throughput-starvation WEDGE (oversubscribed: lower
n_concurrent_trials / raise max_num_seqs) — this was the original rl-131k-cpdcp2r3 silent death.
Serving-throughput LUT — H100-80GB SXM, vLLM, batched continuous-decode (the RL rollout regime). These
are order-of-magnitude health FLOORS, not benchmarks — the diagnostic is the SHAPE (throughput > 0,
scales with Running, Waiting ≈ 0), not hitting a number. Decode is HBM-bandwidth-bound (H100 ≈ 3.35 TB/s);
single-stream ≈ HBM_BW / (2 × active_param_bytes); aggregate rises with batch until compute-bound.
| Model (active params) | Single-stream decode | Healthy aggregate / engine (dozens of Running) | "Awful" — investigate |
|---|
| Dense 7–8B | ~90–140 tok/s | ~2,000–6,000 tok/s | < ~500 tok/s w/ many Running |
| Dense 14B | ~60–100 tok/s | ~1,500–4,000 tok/s | < ~400 |
| Dense 32B | ~30–55 tok/s | ~800–2,000 tok/s | < ~200 |
| MoE 30B-A3B / 35B-A3B (~3–3.5B active) | ~80–160 tok/s | ~2,000–6,000 tok/s (low active → high) | < ~500 tok/s |
| MoE ~235B-A22B (~22B active) | ~25–45 tok/s | ~700–1,800 tok/s | < ~200 |
enforce_eager: true (CUDA graphs OFF) divides decode by ~3–10× (kernel-launch-bound at small decode
batch) — this is exactly why the CoreWeave MoE arms read 15–75 tok/s and the supervisor called it
"awful": the fix was generator.enforce_eager: false (CUDA graphs) in the iris config, NOT a serve-geometry
change. So: if you measure floor-or-below throughput, FIRST check enforce_eager in the resolved config
before blaming weights/geometry. Long context (131k) lowers steady-state tok/s (KV pressure) but should not
floor it. TP/EP geometry: more, smaller engines (e.g. TP=2+EP=2 ×16) should raise aggregate cluster
throughput vs few big TP=8 engines — if it doesn't, suspect the EP/R3 path.
B-training (optimizer/update stage)
- Not VRAM-OOM: no
CUDA out of memory / OutOfMemoryError in logs; memory.used not pinned at
memory.total (≈80 GB) while progress is stalled. (Transient near-ceiling at peak activation is normal
if steps advance.)
- Not RAM/host-OOM: no
OOMKilled in pod state
(kubectl get pod <p> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'), no
oom-killer / Killed in the rank logs, no repeated pod restarts.
- GPUs actually computing: during a step the policy/ref ranks show high
utilization.gpu + power draw
(a forward/backward is compute-bound), not 0%. All-0% with no log advance during "training" = wedge.
Gate B verdict: an engine resident-but-0%/no-tokens, throughput floored-or-zero with Running>0 and
enforce_eager:false, Waiting≫Running flat, or a training-stage OOM / all-ranks-0%-no-progress → lean
KILL (carry evidence to §5). All engines generating ≥ floor with Waiting≈0, or training steps advancing
without OOM → healthy on Gate B, proceed to Gate C.
4. Gate C — Rollout quality (read the actual trace_jobs; use judgment)
State + GPUs can be green while the run produces garbage (e.g. a degraded FSDP→vLLM weight-sync serving a
policy that emits token-salad → 100% reward-0 → no learning signal). So read the literal rollouts under
~/Documents/experiments/traces/<slug>_<stamp>/trace_jobs/. (peek_rl_rollouts.sh <substr> cat <trial-dir> /
grep <regex> also read R2 directly for spot checks.) Work through, using your best judgment — this gate is
qualitative:
- Are trials INITIALIZING? trial dirs being created (config/prompt present). Zero new trial dirs while
engines generate ⇒ the Harbor RolloutCoordinator isn't dispatching (look for
TerminalBenchGenerator initialized … Concurrent trials: K in the log).
- Are trials COMPLETING? count
result.json (the completed-trial marker carrying the reward). At +15/30
min a 131k arm legitimately has ZERO completed (long episodes) — report that as "rollouts executing, 0
completed yet," never as "healthy/done." Completing at a steady rate ⇒ the loop is closing.
- Are any rewards NON-ZERO?
grep result.json for "reward". All-zero / all-timeout is the headline
failure mode — then ask why:
- Agent output is incoherent (token-salad, repetition loops, wrong-language, empty) ⇒ serving/weight-sync
or geometry fault (the FSDP→vLLM sync or the vLLM-fork build for this model/geometry). This is a KILL
signal on a new/untested geometry — the policy being served is not the trained policy. (Check the
tokenizer is right first — but a correct tokenizer + salad output = sync/build, per the prior CoreWeave
diagnosis.) Known CoreWeave MoE cause to check first: the FusedMoE
w13 gate/up swap not re-applied
on the disaggregated RL update (H100/FlashInfer-CUTLASS) → confirm SKYRL_W13_RELOAD_BRACKET is on
(default 1) and the engine log shows finish_weight_reload (fix MarinSkyRL 2bb70a88; marinskyrl doc).
- Agent output is COHERENT but wrong / runs out of turns ⇒ tasks genuinely hard or the harness/verifier
mis-set — NOT necessarily a kill; this can be a real (if low) learning signal. Read several conversations.
- Every trial ends in an environment/infra exception (
VerificationNotCompletedError everywhere =
Daytona sandbox never came up / DAYTONA_* not forwarded; Bearer token invalid = auth) ⇒ infra, not
model — KILL + fix the infra, relaunch.
- Are TURNS completing? in
conversation, count role=="assistant" turns. avg_turns ≈ 1 (agent makes
one move then stops/errors) is the dead-engine / broken-loop signature; multi-turn = real agent behavior.
- Are the AGENT OUTPUTS REASONABLE? actually read 3–5 trajectories: is the model issuing sensible tool
calls / code, or looping / emitting garbage / ignoring the task? (The task framing for the active CoreWeave
arms —
pymethods2test etc. — is simple code-contract Python; it should NOT need massive context, so a
context-overflow storm on those is a red flag, not task difficulty.)
- Are the TASKS too hard / the VERIFIERS failing? sample
verifier_output: is it scoring a genuine
attempt as fail, or erroring/timing out itself (VerifierTimeoutError)? A verifier that never returns a
real score ⇒ no learning signal even with good generations.
Gate C verdict: incoherent generations / all-reward-0 from a serving-or-sync fault on a new geometry, or a
verifier/infra path that yields zero learning signal with no transient explanation → lean KILL (it
cannot learn in this state). Trials completing with some non-zero rewards (or coherent multi-turn attempts on
genuinely-hard tasks even at low pass-rate) → NO-KILL, healthy/learning.
5. Deliver ONE recommendation (the whole point)
Emit exactly one verdict to the supervisor, in this shape:
RL-JOB-HEALTH — /benjaminfeuer/<job> (<model>, <geometry>, <stage>) captured: <traces dir>
VERDICT: KILL | NO-KILL confidence: high|medium|low
Restarts: <B/K burned, remaining K−B> — <none | same failure each attempt: … | transient, recovered>
Gate A (liveness): PASS|FAIL — <state-poll verdict + log-freshness + any wedge/death signature>
Gate B (resources): PASS|FAIL — <per-engine util/mem; aggregate tok/s vs LUT floor; Waiting/Running; enforce_eager; OOM?>
Gate C (rollouts): PASS|FAIL — <trials started/completed; reward distribution; turns; output coherence; verifier sanity>
REASONING: <2–4 sentences — the load-bearing evidence, esp. for whatever was "new/untested" in this run>
NEXT STEPS (if KILL): <root cause + the concrete fix — config knob / weight-sync / image rebuild / infra —
and whether to relaunch on the corrected setting or hold for the supervisor's call>
NEXT STEPS (if NO-KILL): <what to watch next sweep + the specific signal that would flip it to KILL>
Verdict rules:
- KILL if ANY gate is a hard FAIL with no transient/benign explanation: terminal/wedged (A); engines
resident-but-not-generating or floored throughput w/
enforce_eager:false or training-OOM (B); incoherent
generations / all-reward-0 from a serving/sync/verifier fault on a new geometry (C). Give the root cause +
fix — a KILL recommendation without a "what to change before relaunch" is incomplete.
- KILL (restart-burn corroboration, from §0) if the job has burned restarts repeating the SAME failure
each attempt — it is deterministically doomed, and the remaining retry budget will only burn more
nodes-hours reproducing it. State
B/K burned, remaining K−B, same failure: <traceback> and the fix that
must land before any relaunch. (A restart burned on a genuine transient the run has since recovered from is
NOT a kill — say so and weigh the other gates.)
- NO-KILL if all three gates pass, OR the only failures have a legitimate transient/early-bring-up
explanation (e.g. 0 completed trials at +15 min on a long-episode arm; an HF-weight-resolution flake that the
--max-retries/retry-wrapper is catching; gang still admitting). Say what you're waiting on and the signal
that would change the call.
- Default to NO-KILL + escalate on genuine ambiguity (low confidence) — the asymmetry favors not killing a
possibly-healthy bring-up. Hand the supervisor the evidence and let them decide.
You never run the kill. If KILL, the supervisor executes iris job kill /benjaminfeuer/<job> (CoreWeave) /
scancel <id> (SLURM) with permission, then relaunches per rl-agentic-launch-iris / rl-*-launch-* on the
corrected setting. Log the probe + verdict to ~/Documents/agent_logs/ (dated) so the diagnosis isn't lost.
Operating notes
- This skill is for the HARD per-job read;
monitor-cron-sweep is the breadth pass. Don't duplicate the
full sweep here — probe the ONE job you were handed, deeply, and return a verdict.
- Cluster-agnostic by design: the four gates (liveness / resources / rollouts / verdict) hold on any
cluster; only the mechanics (capture tool, state-poll, GPU-poll, log paths) differ — and those live in
.claude/ops/<cluster>/. CoreWeave is the worked example because that's where the new/untested RL runs.
- Read logs through the dependency docs.
.claude/projects/{marinskyrl,vllm,harbor,daytona}/ define the
log vocabulary, the benign-vs-fault line, and the known failure modes — use them so you don't misread a
heartbeat as a hang or a passthrough exception as a crash.
- Never patch/hand-edit on a cluster. If the fix is a config/code change, it goes in the LOCAL clone →
commit → (push / next-launch upload) per CLAUDE.md — your job here is diagnosis + recommendation, not a
cluster-side edit.
