gpu-lock

name	gpu-lock
description	Use when about to launch any GPU/CUDA/PyTorch/Ollama workload, when wrapping a program that uses an LLM or image generator, when handling GPU coordination on Mandragora, or when working with the bots (im-gen, llm-via-telegram, crush, MCP server). Loads the full Mandragora gpu-lock contract — non-blocking respect-the-holder protocol, CLI verbs, library API, VRAM cleanup discipline, hard rules.

gpu-lock — GPU coordination on Mandragora

This machine has one GPU (RTX 5070 Ti, 16 GB). Workloads assume exclusive access. The gpu-lock primitive enforces cooperative serialization with respect-the-holder semantics: a running holder is never interrupted; new arrivals fail fast and the caller decides what to do (retry, fall back, surface the busy state to the user).

When this applies

Anything that touches CUDA: PyTorch, TensorFlow, diffusers, Ollama clients (crush, llm-via-telegram, MCP server), nvidia-smi-monitored work, model loading, inference, fine-tuning, training, profiling.

Two ways in

CLI (any language):

gpu-lock run --name <yourname> --expect <seconds> [--wait <seconds>] -- <cmd>

Tries the fcntl mutex on /dev/shm/gpu-lock/gpu.lock non-blocking. If free: runs cmd, releases on exit. If held: exits 75 (EX_TEMPFAIL) and prints {"error":"gpu-busy","holder":{...}} JSON to stderr. The running holder is never signalled. Add --wait <seconds> to poll (backoff) for a free lock instead of failing fast.

Python library (in-tree projects):

from gpu_lock import gpu_lock, GpuBusy
import torch

try:
    with gpu_lock.acquire("yourname", expected_seconds=N):
        for step in work:
            do_step(step)
        torch.cuda.empty_cache()
except GpuBusy as busy:
    print(f"GPU held by {busy.holder['name']}, ~{busy.expected_remaining():.0f}s remaining")

PYTHONPATH must include /etc/nixos/mandragora/.local/share/gpu-lock for the import to resolve. Mandragora-managed services already set this; ad-hoc scripts can either set it themselves or just use the CLI wrapper.

acquire/acquire_async also accept an optional on_release callback (sync, or awaited in the async form) that runs just before release — a convenient home for the torch.cuda.empty_cache() / evict_model(...) cleanup. Best-effort, not SIGKILL-proof.

Hard rules

1. PyTorch users MUST torch.cuda.empty_cache() before exit. Without it, the caching allocator keeps the pages and the next holder sees a near-full GPU. Real incident on 2026-04-27: a Flux render left 13.7 GiB of cached pages, forcing every subsequent Ollama call into CPU offload mode (2+ minute response times until im-gen restarted).
2. Respect execution. A running holder is not preempted, signalled, or killed. If acquire raises GpuBusy, your job is to back off — retry later, fall back to CPU, or tell the user. Do not os.kill the holder PID and do not loop tight on acquire.
3. No sys.path.insert("/home/m/Projects/gpu-lock") — that path is a bridge symlink for legacy callers and will go away in Phase 2 of the migration. Use PYTHONPATH or the CLI wrapper.
4. Ollama callers acquire the lock themselves; no proxy. llm-via-telegram, local-ai-mcp-server, and vtag wrap every /api/chat|/api/generate POST in gpu_lock.acquire_async(...) plus evict_model(..., keep_alive=0) in the finally. Any new Ollama client must do the same — there is no Layer-2 proxy and none is planned. Bare crush invocations against 127.0.0.1:11434 from one-off shells are the only remaining bypass; treat that as user-supervised, not a contract violator. Pinning keep_alive=-1 is forbidden — a SIGKILL/oomd skips your finally and leaks the entire model into VRAM until next reboot. Use a bounded value (e.g. "10m") so the leak self-clears.

Inspection

• gpu-lock status [--json] — current holder (pid, name, held_for, expected_remaining); --json emits {"holder": {...}|null}
• State files in /dev/shm/gpu-lock/ (RAM-backed, auto-clears on reboot): gpu.lock (the fcntl mutex), gpu.lock.holder (JSON)

Long form

Full design rationale, storage layout, idiomatic patterns, what the tool deliberately does NOT do: /etc/nixos/mandragora/docs/gpu.md.