reserve-gpu

name	reserve-gpu
description	Reserve Iris-backed CoreWeave H100 or GB200 nodes with dev_gpu.py. Use for interactive GPU debugging, multi-node tests, or reconnecting to a dev GPU session.

Dev GPU

Use a dev GPU for short interactive tests. Move to a full Iris job after the code works on the reserved node.

scripts/iris/dev_gpu.py submits a holder job through Iris and uses kubectl exec internally to open a shell. It does not sync files or provide SSH access.

Choose a cluster

• H100: cw-rno2a or cw-us-east-02a
• GB200: cw-us-east-08a

The cluster YAML files under lib/iris/config/ are canonical. If an allocation is slow, or before requesting several nodes, inspect the live backend and queue:

uv run iris --cluster=<cluster> rpc controller list-backends
uv run iris --cluster=<cluster> rpc controller get-scheduler-state

Reserve and connect

Use a unique session name:

export GPU_NAME="${USER}-$(git rev-parse --abbrev-ref HEAD | tr '/' '-')"
export GPU_CONFIG=lib/iris/config/cw-us-east-02a.yaml

uv run scripts/iris/dev_gpu.py --config "$GPU_CONFIG" --name "$GPU_NAME" allocate
uv run scripts/iris/dev_gpu.py --name "$GPU_NAME" connect
uv run scripts/iris/dev_gpu.py --name "$GPU_NAME" status
uv run scripts/iris/dev_gpu.py --name "$GPU_NAME" release

allocate blocks until it is interrupted. An agent running it in the background must call release from another shell. Session state is stored under ~/.cache/marin/dev_gpu_iris/.

The default is one whole H100 node. Pass --gpu-variant GB200 for one GB200 tray. Use --nodes N only when the test needs distributed topology; connect to each pod with connect --node N. Multi-node sessions require whole nodes.

Run uv run scripts/iris/dev_gpu.py <subcommand> --help for current options and defaults.

Sharp edges

• A session holds an expensive node until release. Confirm the holder job is gone if cleanup reports an error.
• connect uses the pod names saved at allocation. If Iris reschedules a task, release and reallocate the session.
• Kubernetes access uses ~/.kube/coreweave-iris; each cluster YAML selects its context. Do not start or restart a controller to fix access without explicit user approval.
• The task image may need GPU JAX installed with cd /app && uv sync --all-packages --extra=gpu. This has been exercised on H100; verify it on GB200.
• Use the pod's MARIN_PREFIX for durable data and rigging.filesystem.marin_temp_bucket(...) for disposable data. Do not read from GCS on CoreWeave without explicit user approval because it can incur egress charges.
• Kubernetes access and object-storage access use different credentials. Task pods receive storage credentials from iris-task-env; dev_gpu.py does not copy the checkout's .marin.yaml. Never print secret values or dump the pod environment.

For cluster access, storage credentials, or stuck pods, read lib/iris/OPS.md and lib/iris/docs/coreweave.md.