rl-agentic-launch-jupiter

name

rl-agentic-launch-jupiter

description

Launch / relaunch agentic RL (SkyRL terminal_bench + Harbor + Daytona) on JSC Jupiter (GH200). Covers the dense 8B/32B FSDP2 arms (seqnorm, TIS, shaped, symclip, lrboost, loopshape) and the MoE/80B Megatron arms (Qwen3-Coder-30B-A3B, Qwen3-Next-80B-A3B) — the exact `python -m hpc.launch --job_type rl` flag set, which flags vary per arm (config / model_path / train_data / num_nodes), runtime+SIF selection, the Daytona RL-org + chain-restart conventions, and the standing constraints (≤6 RL/cluster, a3 CONCLUDED, TIMEOUT restarts are normal). Use when asked to launch / relaunch / refill an agentic SkyRL RL run on Jupiter. Reference: notes/ot-agent/rl_experiments.md, .agents/ops/jupiter/{ops.md,ENVIRONMENT_MAP.md}.

⚠ Do not add comments to YAMLs. Report your recommendations directly to the supervisor.

rl-agentic-launch-jupiter

⚠ Local clone = ground truth (CLAUDE.md §Always). ALL code/config/sbatch edits (OpenThoughts-Agent + MarinSkyRL) go in the local Mac checkouts → commit → push → git pull on the cluster. NEVER hand-edit, git commit, or leave divergent/ untracked changes on a cluster; no patch-by-rsync (vLLM is the only exception — built from source per-cluster). Bake this into every subagent you dispatch.

Agentic SkyRL/GRPO RL runs through python -m hpc.launch --job_type rl with FSDP2 or Megatron. Each rollout is a Harbor agent episode in a Daytona terminal_bench sandbox with a colocated vLLM engine. Jupiter nodes have four 96GB GH200 GPUs. Read .agents/ops/jupiter/ops.md first; runtime/SIF details are in .agents/ops/jupiter/ENVIRONMENT_MAP.md.

1. The canonical launch

🚧 SUBMIT FROM THE REPO DIR WITH DCFT SET. Before launching/resuming: cd /e/scratch/jureap59/feuer1/OpenThoughts-Agent && export DCFT=$PWD (the ops.md preamble does this). The generated universal_rl.sbatch resolves WORKDIR from DCFT_PRIVATE → DCFT → $PWD; submitted from $HOME/a scratch subdir with DCFT unset, the guard detects the wrong dir (missing hpc/shell_utils/triton_cache.sh marker) and exit 1s with FATAL: WORKDIR=... is not the OpenThoughts-Agent repo root. Fix: cd to the repo, export DCFT=$PWD, resubmit.

python -m hpc.launch --job_type rl \
  --rl_config ./hpc/skyrl_yaml/jupiter/<cfg>.yaml \
  --model_path <hf-or-local-model> \
  --train_data '["<HF-repo-or-/abs/task/dir>"]' \
  --num_nodes N \
  --time_limit 11:59:00 \
  --max_restarts K \
  --reservation reformo \
  --experiments_dir /e/data1/datasets/playground/ot-baf \
  --job_name <name>

Varies per arm: --rl_config, --model_path, --train_data, and --num_nodes (§2). Fixed on Jupiter:

• --time_limit 11:59:00 — booster QOS caps walltime at 12h; chain with --max_restarts (§5).
• --reservation reformo — jureap59 booster QOS is suspended (InvalidQOS); reformo is the runnable account/reservation.
• --experiments_dir /e/data1/datasets/playground/ot-baf — the ot-baf personal data root (/ot is read-only-for-you).
• --train_data is a JSON-list string '["..."]' — an HF repo (DCAgent/…, laion/…, SankalpKJ/…) or a pre-extracted local task dir (/e/scratch/jureap59/feuer1/tasks/<name>).
• --job_name <name> — set explicitly for predictable chain-restart and cleanup paths.
• --skyrl_override '++a.b.c=val' — appends a Hydra override (last-wins over the base yaml). For per-arm tweaks without forking a config: sampling (generator.sampling_params.temperature=1.0, …top_p, …top_k, …min_p), Harbor sandbox sizing (++terminal_bench_config.harbor.override_{cpus,memory_mb,storage_mb}), context bumps (++generator.engine_init_kwargs.max_model_len=…). Pass ++-prefixed, struct-safe keys — a bare top-level key risks a Hydra ConfigKeyError.
• Launch from the otagent conda env (/e/scratch/jureap59/feuer1/miniforge3/envs/otagent/bin/python), NOT the RL venv — task extraction imports google.cloud.storage, which the RL venv lacks. (The launcher then selects the RL venv/SIF for the training — §3.)

2. Config map + node count (num_nodes MUST match the config)

num_nodes = GPUs / 4. Pick the config, then set --num_nodes to its budget:

Config (hpc/skyrl_yaml/jupiter/…)	Model	GPUs → --num_nodes
56GPU_seqnorm_tis.yaml (+ extra/56GPU_seqnorm.yaml, extra/56GPU_seqnorm_tis_shaped.yaml)	dense 8B	56 → 14
extra/56GPU_seqnorm_tis_untrunc_symclip.yaml	dense 8B (symclip)	56 → 14
extra/56GPU_seqnorm_tis_untrunc_symclip_loopshape.yaml	dense 8B (symclip+loopshape)	56 → 14
extra/56GPU_seqnorm_tis_untrunc_lrboost.yaml	dense 8B (lr-boost)	56 → 14
56GPU_shaped.yaml (extra/24GPU_shaped.yaml)	dense 8B (shaped reward)	56→14 / 24→6
24GPU_base_131k.yaml / extra/24GPU_base_old.yaml	dense 8B	24 → 6
64GPU_base_32b.yaml, extra/64GPU_base_32b_fp8.yaml, extra/48GPU_*_32b.yaml, extra/128GPU_base_32b.yaml	dense 32B	64→16 / 48→12 / 128→32
24GPU_qwen3_coder_30b_a3b.yaml	Qwen3-Coder-30B-A3B (MoE)	24 → 6
extra/128GPU_qwen3_next_80b_a3b.yaml	Qwen3-Next-80B-A3B (MoE, prod)	64 → 16 (name is historical; header = 64 GPU/16 node)
extra/16GPU_mixtral_8x7b.yaml	Mixtral-8x7B (MoE bring-up)	16 → 4

General rule: 24GPU→6, 48GPU→12, 56GPU→14, 64GPU→16, 96GPU→24, 128GPU→32. The CLI controls -N despite generated #SBATCH --nodes=1. For an unexplained <15-minute failure, check node count first.

3. Runtime / SIF selection

The launcher selects the training runtime (hpc/sbatch_rl/universal_rl.sbatch). Confirm it from the rendered sbatch, rather than assuming.

• Dense 8B/32B FSDP2 (seqnorm / TIS / shaped / symclip / lrboost / loopshape) → RL venv $WORKDIR/envs/rl (torch 2.9). Default RL runtime.
• MoE — Qwen3-Coder-30B-A3B and prod 80B Qwen3-Next-80B-A3B (R3+TIS) → SIF skyrl_megatron_vllm_r3baked.sif (torch 2.9, overlays baked in).
• torch≥2.10 / DCP / torch-native CP / Mixtral-multinode → SIF skyrl_megatron_vllm0202rc0_r3.sif (torch 2.11); stack the skyrl_titan_overlay.img when torchtitan-0.2.2 / _StridedShard (CP+EP) is needed.

Use torch, not vllm.__version__, to identify the runtime. See ENVIRONMENT_MAP §4 for probes and SIF gotchas.

4. Agentic infra conventions

• Daytona uses the RL-org key for RL rollouts (distinct from the eval-org key); set by the launch preamble / hpc/dotenv/jupiter.env, not the CLI.
• Pinggy is EVAL-only, not RL — --pinggy_persistent_url / --pinggy_token are eval-path flags.
• enable_db_registration: false — the launcher auto-injects ++trainer.enable_db_registration=false for RL. Do NOT also pass a bare --skyrl_override enable_db_registration=false (Hydra struct ConfigKeyError risk, redundant). DB registration is a manual cleanup step, not a launch flag.
• Daytona snapshots: a new task set builds snapshots on first launch; caps are HARD (10 new / 60 org). At the org cap, clean stale snapshots first; do not raise the cap: python scripts/daytona/daytona_snapshot_manager.py --api-key-env DAYTONA_RL_API_KEY --delete-stale --yes (deletes only idle/unprotected harbor__* envs — safe; threshold in .agents/projects/daytona/daytona.md). Only a single dataset legitimately needing

  max_new_snapshots unique envs escalates → ask.

• vLLM DP>1 (ray backend): never hardcode --data-parallel-address 127.0.0.1 — Ray registers the head only under its real IPv4 → 127.0.0.1 gives AssertionError: DP master node missing or dead. hpc/vllm_utils.py VLLMServer.start() auto-injects the head IP for DP>1; don't add the flag to new yamls. If overriding, use the real Ray head IPv4.
• MoE / 80B placement: the MoE configs carry their own FSDP/EP sizing in-yaml (Coder-30B: EP=4×FSDP=4=16 policy GPUs + 4 TP=2 vLLM engines = 24 GPU/6 nodes; 80B: 8 TP=4 engines + 8-node FSDP shard = 64 GPU/16 nodes). The 80B yaml sets policy_strict_spread_pg: true (opt-in anti-affinity reserving the policy PG up front to dodge the two-PACK-PG init-OOM race); leave as-configured. Honor the MoE FSDP/EP divisibility constraint (fsdp_size must divide num_experts // ep_size) — don't hand-edit node/EP counts. Details → .agents/projects/marinskyrl/marinskyrl.md.

5. Chain-restart (--max_restarts K)

--max_restarts K submits a head job + K afterany-dependent restart links. A link that hits the 12h wall TIMEOUT auto-resumes from the latest checkpoint in the next link — TIMEOUT is the NORMAL terminal state of a healthy chain, not a failure. Typical K = 5–6.

• A fresh python -m hpc.launch with the SAME --job_name forks to <dir>_2 at step 0 if the original exp dir's configs/*.json exists (the dedup resume-manager engages only for datagen/eval, not RL). To resume instead of forking: either resubmit the existing generated sbatch (experiments/<dir>/sbatch/*_rl.sbatch) via --dependency=afterany, or move the original configs/*.json aside so dedup lands on the un-suffixed dir. (--dry_run regenerates that config → re-move after a dry-run, or skip it.)
• Relaunching auto-resumes from checkpoints/global_step_N/. For a clean ablation, remove <exp>/<job>/<job>/checkpoints/ before relaunching; retain it for a chain extension.
• Always scancel the previous failed/superseded chain before resubmitting.

6. Standing constraints (do NOT violate)

• Daytona RL concurrency ≤ 6 RUNNING per cluster (PENDING restart links don't count). Don't launch a 7th concurrent RL job on Jupiter.
• The a3 series is CONCLUDED — do NOT launch, refill, or auto-advance a3 rows (binary reward + RLOO-n + token_mean; uninformative). Successor arms = the seqnorm / TIS / shaped / symclip / loopshape ablations above. (Exception: DCAgent/r2egym-patched-full-oracle is a separate snapshot-optimized variant — not the a3 row — and launches fine.)
• Never alter config/hparams mid-series. A controlled ablation needing a change → propose a separate experiment; don't mutate the in-flight arm.
• TIMEOUT restarts are expected (§5) — don't treat a chain's TIMEOUT links as failures or salvage them.

7. After launch

• Monitor: monitor-cron-sweep (entropy / log_ratio / grad_norm are mandatory progress columns).
• On completion → rl-agentic-job-cleanup (best-ckpt selection, HF upload from the login node, the manual Supabase DB registration, trace export + parse_skyrl_metrics). enable_db_registration stays false at launch (§4).
• Behavior analysis: analyze-rl-behavior for a post-hoc arm comparison.