babysit-job

name	babysit-job
description	Monitor/babysit a job continuously and recover on failure. Use when asked to babysit, monitor, or watch a job, pipeline, workflow, or training run.

Skill: Babysit Job

Monitor a job continuously and recover on failure. For Zephyr pipelines, delegate to babysit-zephyr instead. Otherwise, follow this skill — Iris is the execution backend.

Ported from marin-community/marin/.agents/skills/babysit-job/SKILL.md with MarinFold-specific adaptations. The marin-MCP-babysitter integration section from the upstream skill is dropped (MarinFold does not currently bundle marin-mcp-babysitter); plain Iris CLI monitoring is fine for our scale.

Required Info

1. job_id — Iris job ID in canonical format /<user>/<job> (e.g., /timodonnell/iris-run-train_protein_30m-20260512-185630).
2. config — Iris config path. MarinFold does not vendor iris configs into the repo — point at whichever YAML file the user maintains for the cluster they want to run on. The marin repo (or the marin-iris wheel) ships example configs at lib/iris/examples/<cluster>.yaml. Common shorthands:
   • marin / marin_prod → lib/iris/examples/marin.yaml
   • marin_dev / marin-dev → lib/iris/examples/marin-dev.yaml
   • coreweave → lib/iris/examples/coreweave.yaml
3. resubmit_command — exact Iris submit command for resubmission; must include --no-wait.
4. For MarinFold TPU training jobs, use --extra tpu (the experiment's pyproject.toml declares the [tpu] extra; see e.g. experiments/exp0_models_protein_docs_initial_port/pyproject.toml).
5. For TPU jobs, the resubmit command must request TPU resources with --tpu <variant>. --reserve <variant> only holds capacity; it does not attach TPU devices to the task container.

Example resubmit command:

uv run iris --config <CONFIG> job run --no-wait \
    --extra tpu --tpu v5p-8 --zone us-east5-a \
    -- python -m train_protein_30m_distance_masked

If any required field is missing, ask for it before proceeding.

Scope

• Recovery is stop then resubmit at the job level.
• Cluster-level actions are out of scope. Do not restart, recreate, or otherwise mutate the cluster unless the user gives explicit consent in the current thread.
• For TPU bad-node errors, escalate to debug-tpu (not yet ported).

Run-history integration

If this is the first time you see a W&B URL for the job, create a run history file:

python scripts/history.py new \
    --wandb-url <URL> --wandb-name <NAME> \
    --experiment <exp<N>_<kind>_<name>-or-no_experiment> \
    --kind <models|evals|data|document_structures|other> \
    --short "<one-line description>" \
    --iris-jobs <JOB_ID>

On every restart (step 7 below), append the new iris job ID:

python scripts/history.py add-iris-job <run-stem-or-wandb-name> <NEW_JOB_ID>

See history/README.md for the policy.

Monitoring Ownership and Duration

• Assign a single monitoring owner when the loop starts.
• Keep this loop running until one of the following:
  • the job reaches a terminal state and the user has acknowledged next action
  • a user-specified stopping point is reached
  • an unrecoverable error is found and reported to the user
• Do not stop early after seeing first loss lines, first eval, or first W&B link.
• Expect monitoring to commonly take 4-5 hours for v5p-8 / v5p-32 protein-LM runs.
• If handoff is needed, transfer ownership explicitly with: current job_id, latest error/signal, W&B link(s), resubmission metadata, and the history/runs/*.md filename if one exists.

Cadence and Tooling Notes

• Cadence default after startup stabilization is sleep 570.
• Startup stabilization sequence (after submit/resubmit):
  • once the job is submitted, sleep 120 and check for immediate failure
  • if still alive, switch to the normal 570 cadence
• Tool-runtime workaround:
  • keep one long-running monitor process/session
  • poll the same session in ~30 second chunks as needed by tool runtime limits
  • repeated no-output polls are expected while waiting for the next 570-second check
• Single monitor process rule:
  • run only one active monitor loop per job to avoid duplicate SSH tunnel and port-binding conflicts
• Sleep must be foreground (max ~10 min due to tool timeout).
• Loop control is at agent level, not bash.

State File

Write to scratch/<create_timestamp>_monitoring_state.json, create the scratch directory if needed. <create_timestamp> should have format YYYYMMDD-HHMM. Track restart_count to detect flapping. State file allows resume after context reset.

{
  "ts": <timestamp_ms>,
  "job_id": "<JOB_ID>",
  "config": "<IRIS_CONFIG_PATH>",
  "wandb_url": "<URL>",
  "wandb_name": "<NAME>",
  "history_file": "history/runs/<YYYYMMDD>_<experiment>_<wandb_name>.md",
  "resubmit_command": "<IRIS_JOB_RUN_COMMAND_WITH_NO_WAIT>",
  "restart_count": 0
}

scratch/ is gitignored.

Loop

1. SLEEP
   - if just submitted/restarted: sleep 120 once
   - otherwise: sleep 570

2. CHECK LOGS
   uv run iris --config <CONFIG> job logs --since-seconds 900 <JOB_ID> | rg -i \
     -e "loss|error|traceback|exception|resource_exhausted|oom" \
     -e "compiler_base\.cc:2587|program hbm requirement|largest program allocations" \
     -e "ownerdiederror|dead node|node death|autoscaler unsatisfied resources" \
     -e "no accelerator found|failed_precondition|device or resource busy"

   `iris job logs <JOB_ID>` includes child-job task logs by default.

3. CHECK STATUS
   uv run iris --config <CONFIG> job list --json --prefix <JOB_ID>

   Terminal success: JOB_STATE_SUCCEEDED
   Terminal non-success: JOB_STATE_FAILED, JOB_STATE_KILLED, JOB_STATE_WORKER_FAILED, JOB_STATE_UNSCHEDULABLE
   Non-terminal: JOB_STATE_PENDING, JOB_STATE_BUILDING, JOB_STATE_RUNNING

   If `pending_reason` indicates worker scale-up/capacity wait, treat as scheduler
   capacity wait — do not run cluster update/recreate/restart actions. Continue
   waiting on cadence, or stop+resubmit only if user explicitly asks.

   Treat RUNNING as controller-level signal only; confirm allocation via expected
   W&B run when possible.

3a. ON TERMINAL STATE / OOM-LIKE SIGNAL — get a structured per-task summary
   (final state, exit, duration, peak memory) instead of grepping logs:

   uv run iris --config <CONFIG> job summary --json <JOB_ID>

   Fast postmortem: e.g. "13/14 shards peaked near the container memory limit
   and failed with exit 137" → cgroup OOM, raise `--memory` on resubmit.

4. PRINT W&B RUN IDS/LINKS (once per training run)

5. CREATE / UPDATE HISTORY FILE
   - First time seeing the W&B URL: `python scripts/history.py new ...`
   - Every restart: `python scripts/history.py add-iris-job ...`

6. REPORT PROGRESS (format: ~<current>/<exact_max> steps)

7. EVALUATE (terminal? error? stalled? -> recover or continue)

8. RECOVER (STOP -> RESUBMIT)
   - If current job is still non-terminal, stop it first:
     uv run iris --config <CONFIG> job stop <JOB_ID>
   - Then resubmit:
     <RESUBMIT_COMMAND>
   - Capture `job_id` from output (line like `Job submitted: /<user>/<job>`).
   - Iris nuance:
     - if `resubmit_command` omits `--job-name`, Iris auto-generates a fresh id each resubmission.
     - if `resubmit_command` uses a fixed `--job-name`, Iris may reuse the same id
       after terminal completion by replacing the finished job.
   - Append the new job id to the history file:
     python scripts/history.py add-iris-job <run-stem> <NEW_JOB_ID>
   - Update state file: `job_id=<NEW_JOB_ID>`, `restart_count += 1`.
   - Go to step 1.

Fixing Small Bugs

When EVALUATE detects an error, before recovery:

1. Analyze logs — look for Traceback, Error, Exception. Identify file and line.
2. If small fix (typo, missing import, wrong variable name): fix it, then RECOVER.
3. If complex (architectural, unclear cause, broad investigation): report to user, exit loop.

Small-fix examples: NameError, ImportError, SyntaxError, obvious KeyError.

Complex examples: OOM, TPU/XLA HBM exhaustion, distributed training failures, data loading issues, unclear multi-file stack traces.

Error Patterns

• Treat TPU/XLA HBM reports as failure even without literal OOM:
  • Program hbm requirement ...
  • Largest program allocations in hbm
• If progress stalls across multiple intervals with OwnerDiedError, dead node, or unsatisfied resources -> mark degraded and notify user.
• If same error repeats after one fix attempt, do not retry blindly; report to user.

MarinFold-specific signals

• Cross-region GCS I/O latency: if checkpoint writes look slow, confirm the TPU zone matches the bucket region (us-east5-a for marin-us-east5). See models/AGENTS.md "Co-locate TPU with checkpoint bucket".
• Tokenizer cache poisoning: a sudden vocab_size mismatch usually means the HF tokenizer URL was rewritten without a revision pin. See models/AGENTS.md "Tokenizer revisions are pinned".

When to Escalate

• Debug Zephyr pipeline issues → babysit-zephyr (sibling skill) and then debug-zephyr-job (not yet ported).
• Debug TPU bad-node errors → debug-tpu (not yet ported).
• Debug running tasks with iris task exec → debug-iris-job (not yet ported).

Notes

• Iris job list --prefix requires canonical job names (/<user>/<job>), not short names.
• Iris monitoring is job-level; cluster updates are not part of normal recovery.