gpu-lock-skill

name	gpu-lock-skill
description	Coordinate multiple Codex agents or tasks sharing the same NVIDIA GPU by acquiring a local GPU lease before tests, benchmarks, profilers, or any command that may execute CUDA work; includes stale lease timeout cleanup, soft GPU hiding, and zombie/unknown GPU process checks.

GPU Lock

Use this skill whenever work may run on a shared NVIDIA GPU. The goal is to let multiple agents think, edit, compile, and do CPU-only work concurrently while serializing only the windows that actually use the GPU.

Required Rule

Before running any command that may touch the GPU, acquire a GPU lease with scripts/gpu_lock.py. This includes:

• CUDA tests and benchmarks
• nvidia-smi based validation that depends on current GPU state
• Nsight Compute / Nsight Systems profiling
• GPU-enabled pytest, ctest, make test, python, torch, triton, cuda-gdb, or custom binaries

Prefer the run subcommand so the lease is released automatically:

python /path/to/gpu-lock-skill/scripts/gpu_lock.py run --gpu 0 --timeout 30m --owner "$USER-codex" -- make bench

If the command exits, the lock is released. If the command exceeds the lease timeout, the wrapper terminates the launched process group and reports what it killed.

Every lease also starts a lightweight watchdog process by default. If the owning agent hangs and the lease is still present after expires_at, the watchdog reclaims the lease, kills recorded process groups and same-user GPU processes on that GPU, and writes the action to the lock directory's watchdog.log.

Soft GPU Hiding

For simple access control, start agent shells with GPUs hidden:

source /path/to/gpu-lock-skill/scripts/codex_gpu_env.sh

This sets CUDA_VISIBLE_DEVICES=-1 when no lease is active, so normal CUDA runtime users should report no visible device. It is a cooperative guard, not a security boundary; a process can still override the environment if it is allowed to.

The run subcommand temporarily sets CUDA_VISIBLE_DEVICES only for the wrapped command:

python /path/to/gpu-lock-skill/scripts/gpu_lock.py run --gpu 0 --timeout 30m -- make bench

The parent agent shell remains hidden after the command exits.

Manual Lease

Use a manual lease only when several GPU commands must run under the same lease:

eval "$(python /path/to/gpu-lock-skill/scripts/gpu_lock.py acquire --gpu 0 --timeout 30m --owner "$USER-codex" --format shell)"

Run GPU commands only after acquire succeeds. Release the lease when finished:

eval "$(python /path/to/gpu-lock-skill/scripts/gpu_lock.py release --gpu 0 --token "$CODEX_GPU_LOCK_TOKEN" --format shell)"

The shell-format release unsets CODEX_GPU_LOCK_TOKEN and hides GPUs again with CUDA_VISIBLE_DEVICES=-1.

For long manual sessions, extend the lease before it expires:

python /path/to/gpu-lock-skill/scripts/gpu_lock.py refresh --gpu 0 --token "$CODEX_GPU_LOCK_TOKEN" --timeout 30m

Timeout Guidance

Choose the shortest timeout that covers the expected GPU execution window:

• Kernel smoke test: 5m to 10m
• Unit tests or small benchmark: 15m to 30m
• Profiling or larger benchmark sweep: 45m to 2h

Do not hold the lease while editing code, reasoning, installing dependencies, or compiling CPU-only artifacts.

Stale Lease Behavior

When a lease expires, the next acquire, run, status --cleanup, or cleanup invocation reclaims it. The background watchdog normally reclaims it first; the command-time cleanup path is a fallback in case the watchdog was disabled or killed.

Reclaiming an expired lease:

• removes the stale lock
• terminates process groups recorded by the lock wrapper
• terminates same-user processes currently using the target GPU
• escalates from SIGTERM to SIGKILL if processes do not exit
• prints the affected PIDs/process groups

Unknown GPU processes that do not belong to an existing lock are not killed by default. The command reports them and refuses the lease. If the operator explicitly wants stale unknown same-user GPU processes cleaned, pass:

--kill-unknown-stale-after 30m

Status And Debugging

Check current leases and GPU processes:

python /path/to/gpu-lock-skill/scripts/gpu_lock.py status --gpu 0

Force cleanup of expired leases:

python /path/to/gpu-lock-skill/scripts/gpu_lock.py cleanup

Use --format json when another script needs structured output.

The lock state lives in CODEX_GPU_LOCK_DIR when set, otherwise under XDG_RUNTIME_DIR/codex-gpu-lock or ~/.cache/codex-gpu-lock.

Agent Conduct

If acquiring the lease fails, report the owner, remaining lease time, and current GPU process list to the user instead of bypassing the lock. If a stale lease is reclaimed and processes are killed, include the killed PIDs or process groups in the user-visible update.