datagen-launch-iris

name	datagen-launch-iris
description	Launch, monitor, and manually clean up a trajectory-generation (datagen) job on Marin's Iris TPU cluster via the OpenThoughts-Agent entrypoint. Use when asked to start, watch, rescue, or kill a datagen/tracegen run on Iris.

datagen-launch-iris

📍 Iris orientation — read first. Read the Iris tools catalog (.agents/ops/iris/ops.md) and the Iris ops directory (.agents/ops/iris/ — ops.md for CoreWeave GPU, ops.md for TPU marin) for binding access/preamble/gotchas and the helper-script inventory. For eval jobs use eval-agentic-launch-iris instead.

Required info (ask if missing)

1. tasks — --tasks_input_path: an HF dataset id (e.g. DCAgent/exp_rpt_e2egit-v2; the launcher snapshot_downloads it and auto-explodes a task_binary parquet into task dirs) or a local tasks dir. Not both.
2. slug — short dataset name for the job name + HF repo (e.g. e2egit-v2).
3. Operating point — default S1 (Qwen3.5-122B-A10B-FP8, 32k, v5p-8, single-host). Don't change configs unless asked.

Prerequisites

• Python 3.12 env (the iris client writes the launcher's sys.version_info into the worker's uv sync --python): source /Users/benjaminfeuer/miniconda3/etc/profile.d/conda.sh && conda activate otagent (or call /Users/benjaminfeuer/miniconda3/envs/otagent/bin/python).
• Secrets: source "$DC_AGENT_SECRET_ENV" (DAYTONA_API_KEY for host-side snapshot pre-build, HF_TOKEN for upload, MARIN_HMAC_* for runai_streamer, OPENAI_API_KEY for LLM-judge datasets — see below). Also pass --secrets-env <path> so they reach the worker. Do not echo values.
• LLM-judge datasets need OPENAI_API_KEY in the TRIAL env (laion/stackexchange-superuser-sandboxes-verified, laion/stackexchange-tezos-sandboxes-verified, and any nemotron_gym llm_judge/*_judge set). Each task's task.toml carries [verifier] env = { OPENAI_API_KEY = "${OPENAI_API_KEY}", JUDGE_MODEL = "${JUDGE_MODEL:-}" }; harbor's verifier resolves that ${OPENAI_API_KEY} from the worker process env (harbor/src/harbor/verifier/verifier.py verify() → resolve_env_vars(merged_env) → environment.exec(env=…)) and injects it into the Daytona verifier sandbox where the litellm/gpt-4o-mini judge runs. The mechanism is already wired: nothing extra to pass — --secrets-env "$DC_AGENT_SECRET_ENV" loads every KEY=VALUE from the secrets file (including OPENAI_API_KEY) into the iris worker's env_vars (hpc/iris/env.py:_load_worker_secrets_env, plus an explicit OPENAI_API_KEY passthrough at _forward_launcher_env), so the launch recipe below (which already passes --secrets-env) propagates the key end-to-end. The ${OPENAI_API_KEY} template is REQUIRED (no default) — if the secrets file lacks it, harbor raises a ValueError at verify time (not a silent 0.0). Just make sure OPENAI_API_KEY is present in $DC_AGENT_SECRET_ENV.
• If a launch fails with marin-iris client is too old, run git -C /Users/benjaminfeuer/Documents/marin pull --ff-only origin main (editable install) — not uv sync.

Launch

cd /Users/benjaminfeuer/Documents/OpenThoughts-Agent
source /Users/benjaminfeuer/miniconda3/etc/profile.d/conda.sh && conda activate otagent
source "${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV to the secrets file first}"
TS=$(date +%Y%m%d-%H%M%S)
python data/cloud/launch_tracegen_iris.py \
  --harbor_config hpc/harbor_yaml/datagen/ctx32k_verified.yaml \
  --datagen_config hpc/datagen_yaml/qwen3_5_122b_a10b_fp8_runai_v5p8_s1.yaml \
  --tasks_input_path <HF-dataset-id | /abs/tasks/dir> \
  --tpu v5p-8 --preemptible \
  --n_concurrent 64 --n_attempts 1 --health_max_attempts 600 \
  --job_name "qwen3.5-122b-32k-<slug>-${TS}" \
  --secrets-env "$DC_AGENT_SECRET_ENV" \
  --upload_hf_repo penfever/<slug>-qwen3.5-122b-32k-traces \
  --no-wait

Flag notes:

• Serve config resolves from model_config/ (commit e792bfbb): the launcher looks up the served model (from --datagen_config's engine.model / --model) in model_config/<org>/<slug>.yaml and merges/forwards its agent_kwargs + applies serve intrinsics (max_model_len/limit_mm/extra_args) on the worker. tp_size + harbor_config are ignored on TPU (tp from chip count; harbor_config CLI-required). Precedence: explicit CLI / --datagen_config values always win; a model with no entry launches byte-unchanged (logged). Edit model_config/<org>/<slug>.yaml, never the generated eval/configs/model_configs.yaml.
• OMIT --gcs-output-dir for the normal path. The launcher auto-pins the job to the region with the most v5p-8 capacity and routes outputs to that region's co-located single-region bucket (gs://marin-us-east5/ot-agent, …; output_bucket_for_region in hpc/iris/regions.py). The registry records that URI, so rescue/analyze resolve it automatically (hpc.iris.job_output_resolver).
• --gcs-output-dir gs://marin-models-us/ot-agent opts OUT of region pin + single-region routing (forces multi-region bucket). Only pass to override — e.g. to escape a stuck-PENDING trap when one region's v5p-8 pool collapsed. Reverts outputs to pricier multi-region; prefer leaving the pin on.
• ctx32k_verified.yaml = verifier ON + release_trial_payloads_in_memory (bounds worker host-RAM so heavy/repo datasets don't OOM). Use the 32k config with the 32k S1 engine.
• --health_max_attempts 600 mandatory (122B-FP8 cold compile ~60 min; default 100 ≈ 50 min kills the job before first serve).
• --n_concurrent 64 = max_num_seqs(64) × DP(1).
• Image :tpu at/after digest ae085bc8 auto-uploads the HF repo on a clean (state-4) completion — no manual rescue needed.

After submit, confirm placement:

/Users/benjaminfeuer/Documents/marin/.venv/bin/iris --cluster=marin query \
  "SELECT job_id, state FROM jobs WHERE job_id='/benjaminfeuer/<job>'" -f csv

state 1=PENDING, 2=starting, 3=RUNNING, 4=SUCCEEDED, 5=FAILED, 6=KILLED.

Monitor

/Users/benjaminfeuer/miniconda3/envs/otagent/bin/python \
  /Users/benjaminfeuer/Documents/OpenThoughts-Agent/scripts/iris/analyze_iris_harbor_job.py \
  /benjaminfeuer/<job> --output /tmp/<job>_history.md --resync

Read the .json sidecar (it paginates full history — don't eyeball --tail): total_runtime_s, iris_preemption_count, cycles[] (each with did_serve/time_to_first_serve_s), serving_summary.gen_tps/.running (n/mean/max), non_empty_trials/total_trial_dirs (productive rate), harbor_exception_stats. S1 baseline ≈ 400 mean / 1115 peak gen tok/s; short-task datasets (nl2bash, e2egit) run lower by nature — judge by productive trial rate, not tok/s alone.

Did it auto-upload? On a state-4 job from image ae085bc8+, check the repo exists before rescuing:

/Users/benjaminfeuer/miniconda3/envs/otagent/bin/python -c \
 "from huggingface_hub import HfApi; print(HfApi().dataset_info('penfever/<slug>-qwen3.5-122b-32k-traces').lastModified)"

Traces → tool-calling SFT. To turn the generated traces into SFT-ready rows (role: tool + structured tool_calls, not the lossy default conversations shape), use harbor traces export --sft-format → .agents/projects/harbor/ops.md § "SFT-ready traces with tool calling — harbor traces export --sft-format". For SFT decoded byte-exact from the served tokens instead (--record_literal jobs, TIS fidelity), see § "Literal-token trace datasets" in the same doc.

Manual cleanup

Kill (only with explicit permission for a RUNNING/placed job):

/Users/benjaminfeuer/Documents/marin/.venv/bin/iris --cluster=marin job kill /benjaminfeuer/<job>

Rescue banked traces (any terminal job whose repo did NOT auto-create — killed, OOM, or pre-ae085bc8 image). Rsync the GCS job dir local, then push:

source "${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV to the secrets file first}"
OUT=$(/Users/benjaminfeuer/miniconda3/envs/otagent/bin/python -m hpc.iris.job_output_resolver <job> \
      --cluster /Users/benjaminfeuer/Documents/marin/lib/iris/config/marin.yaml)
mkdir -p /tmp/<job>_traces
gsutil -m rsync -r "$OUT/<job>/" /tmp/<job>_traces/
/Users/benjaminfeuer/miniconda3/envs/otagent/bin/python \
  /Users/benjaminfeuer/Documents/OpenThoughts-Agent/scripts/harbor/make_and_upload_trace_dataset.py \
  --job_dir /tmp/<job>_traces \
  --repo_id penfever/<slug>-qwen3.5-122b-32k-traces \
  --episodes last --filter none --skip_register

Resolve the RECORDED output prefix via job_output_resolver — never hardcode a bucket. --skip_register = upload only, no Supabase row. Repo is public.

Daytona snapshot cap — if a launch fails SnapshotCapExceeded on the shared cli org, delete ONLY broken (MISSING-state) harbor__* snapshots (never ACTIVE — may belong to running jobs):

source "${DC_AGENT_SECRET_ENV:?set DC_AGENT_SECRET_ENV to the secrets file first}"
/Users/benjaminfeuer/miniconda3/envs/otagent/bin/python - <<'PY'
import os
from hpc.snapshot_manager import _parse_org_arg, _SnapshotManager, list_snapshots
org=_parse_org_arg(f"cli={os.environ['DAYTONA_API_KEY']}")
mgr=_SnapshotManager([org]); client=mgr._client(org)
for snaps in list_snapshots([org]).values():
    for s in snaps:
        if s.name.startswith("harbor__") and s.state=="MISSING":
            client.snapshot.delete(client.snapshot.get(s.name)); print("deleted", s.name)
PY

Do NOT run broad cleanup_unused_snapshots against the shared cli org — it deletes by your task set and would remove teammates' ACTIVE snapshots. When the org is at cap with 0 MISSING (all ACTIVE) but many idle harbor__ leftovers, use the idle-gated reclaim via utils-reclaim-stale-snapshots: daytona_snapshot_manager.py --name-prefix harbor__ --delete-stale at the threshold in .agents/projects/daytona/daytona.md.

Stuck PENDING (no v5p-8 capacity): report it and relaunch UNPINNED (--gcs-output-dir gs://marin-models-us/ot-agent above). Kill the stuck submission only with permission.

Current campaign: 131k A2 opencode (2026-07) — deltas from the 32k S1 default

Campaign specifics (dataset order, keep-3 state, per-arm status) live in the tracker: /Users/benjaminfeuer/Documents/experiments/active/qwen3.5-122b-131k-datagen-opencode-iris/tracker.md. Dated chronology: ~/Documents/agent_logs/2026-07-08_qwen3.5-122b-131k-datagen-opencode-iris_history.md. Literal-trace decode/rescue reference: .agents/projects/harbor/ops.md (§ Literal-token trace datasets). Ingress topology (native /proxy/t/* capability-URL, token TTL, pinggy rollback): .agents/ops/iris/ops.md.

Recipe deltas from the Launch block above:

• --datagen_config hpc/datagen_yaml/extra/qwen3_5_122b_a10b_fp8_runai_v5p8_131k_dp1_tp4_ep1_s32.yaml (A2: TP4/DP1/EP-on/seqs32/131k), --n_concurrent 32.
• --harbor_config hpc/harbor_yaml/datagen/opencode_ctx131k.yaml — FULL path required (a bare opencode_ctx131k.yaml FileNotFoundErrors on the worker — no config-dir fallback).
• --agent opencode --record_literal (literal-token capture; per-serve-unique log filenames so a preempt-resume can't clobber attempt-0's log).
• --max-retries 200 — retry hard task failures. Iris separately tolerates 1,000 infrastructure preemptions per task. Tracegen launches also default --max-task-failures 10, so an isolated hard failure after several preemptions no longer aborts the whole resumable job while repeated failures still stop it. Override the cumulative guard explicitly only for bounded smokes. A resume continues the same GCS serve dir (one serve dir, rising task count) instead of restarting from task 1.
• repo naming penfever/<slug>-qwen3.5-122b-131k-opencode-traces.

MODEL PATH GUARD (the #1 launch pitfall): --model gs://marin-models-us/ot-agent/models/Qwen/Qwen3.5-122B-A10B-FP8/ — the mirror, which streams via runai-streamer direct-to-device (~0 local disk). NEVER the bare HF id Qwen/Qwen3.5-122B-A10B-FP8 — that string is ONLY the uploader's --served_model provenance stamp; passing it as --model defeats streaming → vLLM snapshot_downloads ~120GB → OOMs the 100GB worker disk → startup hang (0 trials for hours, never self-heals). After launch, grep the job's log to confirm the baked command carries --model gs://….

Native ingress (pinggy retired): add --ingress_mode controller --ingress_host https://iris.oa.dev. For the normal Marin TPU datagen path, the worker registers a LINK endpoint and mints locally because its controller is the Marin parent; Daytona receives https://iris.oa.dev/proxy/t/<token>/otagent-<slug>/v1. A CoreWeave-hosted serving job is different: submit it through the Marin meta-scheduler with --target-cluster cw-us-east-02a, wait for the endpoint mirror on Marin, and mint at Marin with the forwarded Marin login. Never send Daytona to iris-cw-us-east-02a.oa.dev or use a CoreWeave-peer-minted token. Route health: curl -sk -w "%{http_code}" https://iris.oa.dev/proxy/t/badtoken/serve.nope/v1/models → 401.

Liveness / wedge check — verify FRESHNESS, not just "activity present". A RUNNING job can sit state 3 with vLLM logs full of running agent lines yet be wedged — every line stamped at the SAME frozen timestamp. Judge by whether things ADVANCE over a short live window: (1) OUTER-dir GCS trial-dir count grows (sample, wait ~4–5 min, re-count); (2) harbor <done>/<total> completed advances / result.json updated_at is fresh; (3) vLLM emits fresh chat/completions 200 OK + nonzero-and-moving Running: N (newest log TIMESTAMP recent, not frozen). Frozen on all three with an engine that served-then-stopped = confirmed wedge → kill+rescue+relaunch (standing kill authority). Carve-out: a job mid harbor GCS-jobs_dir resume scan after a preempt can be legitimately progress-frozen for up to ~6h WITH a recent engine-ready marker and NO prior serving on this attempt — that's NOT a wedge; don't kill (monitor-restore-iris §4c).

Rescue vs RESUME policy (RESUME by default for sub-60% jobs): when a job goes terminal short of completion, default to RESUME (relaunch the same arm → same baked --job_name/gs:// job dir → harbor continues the remaining tasks), NOT harvest a partial banked slice. Rescue (banked-GCS → HF) ONLY when ≥60% complete (<done>/<total> from the harbor Mean: line, or productive-trial dirs / total); below 60% a rescue just publishes a stub and burns the arm's identity.

Harbor resume/export-push fix is NOT deployed in any current :tpu image: the build's uv pip install ".[datagen-tpu]" resolved a cached pre-fix harbor==0.8.0 wheel. Consequence: every preemptible datagen job dies on its first preempt-resume export-push → banked-GCS rescue is the reliable harvester. Real remedy (⚑ user decision): an image rebuild that busts the harbor wheel cache (bump harbor version / HARBOR_COMMIT + --force-reinstall, or fresh-from-GitHub source install), then VERIFY by grepping the installed harbor/job.py INSIDE the built image — see build-tpu-image-iris. Fixing harbor code requires busting the cached wheel to take effect in an image rebuild — the general lesson.

Terminal-job triage — rescue vs BLOCKED (precheck before rescuing):

• Worker in-job auto-upload lands TEXT-ONLY (the pinned :tpu predates the schema-pin fix) OR fails at export-push (harbor FileExistsError on preempt-resume — see caveat above) — both leave banked GCS trials + logs/*_literal.jsonl intact → RESCUE from GCS (see Manual cleanup) with --served_model Qwen/Qwen3.5-122B-A10B-FP8, verify count_populated_literal_rows ≈ correlation yield. "SUCCEEDED"/"repo exists" is NEVER proof of trainable literals — always check the true count.
• A job with 0 valid traces (100% steps:0 / NonZeroAgentExitCodeError exit 127 = agent binary absent in the sandbox) is GARBAGE → mark BLOCKED, NOT rescuable, needs a full re-run after the sandbox-install fix. Spot-check a few trials' result.json/exception.txt before rescuing.

Harbor editable guard: the uploader imports harbor from /Users/benjaminfeuer/Documents/harbor — it MUST be on main (verify + git checkout main && git pull if drifted; penfever/working RETIRED) before any rescue, or the export crashes.

Guardrails

• Kill authority: a HEALTHY (progressing) running job → NEVER kill without explicit permission. A CLEARLY-DOOMED / mis-launched job making 0 progress for hours that won't self-heal (bare-HF-id disk-full hang, exit-127 garbage, or a severed-Daytona zombie — frozen samples AND frozen harbor exception counters) MAY be killed + relaunched correctly WITHOUT asking (standing authorization, 2026-07-07); log it to the tracker.
• NEVER stop/restart/bounce a HEALTHY RUNNING job or the Iris cluster without explicit permission in the current thread.
• NEVER read/write GCS across regions (cost). Keep everything in the bucket-matched US region.
• Rescue + snapshot-MISSING cleanup are safe maintenance; killing a running job and broad snapshot pruning are not — confirm first.