analyze-training-run-iris

name

analyze-training-run-iris

description

Detailed health check for a Levanter/executor TRAINING run on the marin Iris cluster (e.g. the delphi midtraining runs) — step progress vs target, loss/throughput, preemption + MAJOR step-gap detection, and checkpoint cadence. Use for an executor coordinator (`<run>-coord`) plus its nested `<run>-coord/checkpoints-<step>-<hash>` training child, which the harbor analyzer (analyze-job-history-iris) does NOT cover (training has no harbor trial sidecars, same as GPU-RL). Reads W&B per-step history + `iris job summary` + GCS checkpoints instead of harbor GCS output.

analyze-training-run-iris

📍 Iris orientation — read first. Before acting on anything in this skill, read the Iris tools catalog (.claude/ops/iris/iris_tools.md) and the Iris ops directory (.claude/ops/iris/ — the CoreWeave GPU particulars in coreweave_gpu_ops.md, the TPU marin particulars in iris_job_lifecycle.md). They carry the binding access/preamble/gotchas and the helper-script inventory the steps below rely on.

A Levanter training run launched through the marin executor surfaces as TWO Iris jobs:

• a tiny CPU coordinator — /<user>/<run>-coord — the executor_main DAG-walker (it submits the training job and then blocks); and
• its nested training child — /<user>/<run>-coord/checkpoints-<step>-<hash> — the multi-task v5p job where the actual training steps happen (e.g. 8 tasks for a v5p-64).

Health = the CHILD's step progress + the run's preemption/gap history. The harbor analyzer (analyze-job-history-iris) does NOT apply — a training job has no harbor trial sidecars (just like GPU-RL). Use the three sources below instead. W&B is primary for step/loss/throughput; iris job summary is primary for preemptions/liveness; GCS is primary for checkpoint cadence.

Source 1 — iris job summary: preemptions, liveness, per-task state (always available)

IRIS=/Users/benjaminfeuer/Documents/marin/.venv/bin/iris
$IRIS --cluster=marin job summary <child_job_id>

Report preemptions=N failures=N, tasks running/completed (all N tasks should be running together — a v5p job is gang-scheduled), the longest task DURATION, and PEAK MEM. preemptions>0 is expected on a preemptible v5p — each one means iris restarted the slice and Levanter resumed from the last checkpoint (every preemption costs a wall-clock gap: re-place + reload weights + XLA recompile). failures>0, a shrinking task count, or a crash-restart loop is a red flag → read the child logs for the error.

Source 2 — W&B per-step history: step / loss / throughput + MAJOR GAP detection (primary)

The run logs to W&B project delphi-midtraining (entity nyu-dice-lab); the run name is the GCS output-path hash — the last path segment of gs://marin-us-east5/checkpoints/<run>-<hash> (e.g. delphi-1e23-p33m67-k0p20-lr0.67-b6607e → run delphi-1e23-p33m67-k0p20-lr0.67-b6607e). Per-step history is NOT mirrored by mum — query the W&B API directly (needs WANDB_API_KEY from $DC_AGENT_SECRET_ENV; use the otagent python which has wandb):

source "${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV to the secrets file first}"
/Users/benjaminfeuer/miniconda3/envs/otagent/bin/python - <<'PY'
import wandb
ENTITY, PROJECT, RUN = "nyu-dice-lab", "delphi-midtraining", "<run-hash>"   # <-- the ...-b6607e hash
api = wandb.Api()
r = api.run(f"{ENTITY}/{PROJECT}/{RUN}")
total = r.config.get("trainer", {}).get("num_train_steps") or r.config.get("num_train_steps")
h = r.history(keys=["_step", "_timestamp", "_runtime", "train/loss"], pandas=True)
if h is None or len(h) == 0:
    print("state:", r.state, "-> pre-first-step (still setup/HF-download/XLA-compile); no training step yet")
else:
    h = h.dropna(subset=["_step"]).sort_values("_step")
    cur = int(h["_step"].iloc[-1])
    ts = h["_timestamp"].to_numpy()
    deltas = [b - a for a, b in zip(ts[:-1], ts[1:])]
    med = sorted(deltas)[len(deltas)//2] if deltas else 0.0
    thr = max(300.0, 20.0 * med)                       # same MAJOR-GAP rule as compute_time.md
    gaps = [d for d in deltas if d > thr]
    toks = 0
    bs = (r.config.get("trainer", {}) or {}).get("train_batch_size")
    sl = r.config.get("train_seq_len") or (r.config.get("model", {}) or {}).get("max_seq_len")
    if bs and sl and med: toks = bs * sl / med
    loss = h["train/loss"].dropna()
    loss = float(loss.iloc[-1]) if len(loss) else None
    eta_h = ((total - cur) * med / 3600.0) if (total and med) else None
    print(f"state            : {r.state}")
    print(f"step             : {cur}/{total}  ({round(100*cur/total,1) if total else '?'}%)")
    print(f"train/loss       : {loss}")
    print(f"median step dt   : {round(med,2)} s   -> ~{round(toks):,} tok/s" if med else "median step dt: n/a")
    print(f"MAJOR gaps       : count={len(gaps)}  total={round(sum(gaps)/3600,2)} h  (threshold {round(thr)} s)")
    print(f"ETA to {total}   : ~{round(eta_h,1)} h of compute (excludes future preemption gaps)" if eta_h else "ETA: n/a")
PY

• step cur/total = progress against the K-budget target (e.g. 29,945 for K=0.20). This is the headline "progress in steps" number.
• MAJOR gaps = consecutive _timestamp deltas exceeding max(300 s, 20 × median step interval) — preemption / idle gaps (the metric the user wants: "note major gaps"). Cross-check count against iris job summary preemptions — they should be the same order (a gap with NO matching preemption is a silent stall worth flagging). Report gap count + total hours.
• tok/s = batch × seq / median_dt. Compare across ticks; a sudden drop = contention or a bad slice.
• Empty history = the run is still in setup / HF-weight download / first XLA compile — that is NOT a gap; report it as pre-first-step and move on.

Source 3 — GCS checkpoint cadence: resume safety (always available)

gsutil ls gs://marin-us-east5/checkpoints/<run>-<hash>/ | grep -E 'step-[0-9]+' | tail -5
gsutil ls -l gs://marin-us-east5/checkpoints/<run>-<hash>/step-<latest>/ 2>/dev/null | tail -2   # timestamp

Report the latest persisted step + its timestamp (the checkpointer saves on an interval — e.g. save_interval 10m, keep every 1500). The latest checkpoint lagging a bit behind the W&B step is fine (async save). But no step-* checkpoint long after training started is a red flag — under preemption the run would lose all un-checkpointed progress. Only .executor_info / .executor_status* present (no step-*) = still pre-first-checkpoint (early bring-up).

Don't mistake setup steps for training steps

iris job logs <child> early on shows [iris setup] step N/M lines — those are uv-sync SETUP steps, NOT training steps. Do NOT grep step N from the logs for progress. Use the W&B _step (Source 2) as the authoritative training-step counter; only fall back to Levanter's own in-log training-step line if W&B is unreachable.

The compact cron line (one per training run)

<run> state=running step=<cur>/<total> (X%) loss=<L> ~<T>tok/s preempts=<P> gaps=<G>/<H>h ckpt=step-<C> plus a one-line health read: past setup/compile? step rate sane vs the prior tick? loss finite and trending down (not NaN/spiking)? preemptions resuming cleanly (checkpoint advancing)? ETA to the K-budget target.

Running it / offloading to a subagent

The W&B pull is fast (seconds), unlike the harbor analyzer — you usually do NOT need a subagent. If a run is huge or you are sweeping several, the analyze-job-history-iris foreground-and-wait discipline still applies to any slow gsutil/log reads, but the W&B query itself is quick.

Related skills

• monitor-cron-sweep-iris — the every-3-hours sweep; its class E (executor/Levanter training) invokes this skill, just as class A/B invoke analyze-job-history-iris.
• analyze-job-history-iris — the harbor (datagen/eval) analyzer; does NOT apply to training runs.
• rl-job-health-deep-dive — the GPU-RL equivalent (also no harbor sidecars; uses the finelog).