| name | zombie-hunt |
| description | Find zombie processes holding GPU VRAM on lab servers (dead notebook kernels, orphaned trainers) and propose kill commands. Report-only. Use when GPUs look occupied but idle, or for "who is using GPU N". |
Zombie hunt (report-only)
Goal: explain every GPU whose memory is held, and flag holders that are
probably dead weight. You never send a signal yourself.
-
list_hosts → hosts where GPUs are inUse. For each, via run_command:
-
Map memory to processes:
nvidia-smi --query-compute-apps=gpu_uuid,pid,process_name,used_memory --format=csv,noheader
nvidia-smi --query-gpu=index,uuid --format=csv,noheader
-
Inspect each PID:
ps -o pid,ppid,user,etime,stat,%cpu,cmd -p <pid1>,<pid2>,...
-
Zombie heuristics — call it a zombie only on 2+ signals (or 1 signal plus
user confirmation):
- jupyter/ipykernel holding VRAM with ~0 %cpu over a long etime
- orphaned trainer (ppid 1) with 0% GPU util across two samples ~15s apart
- process state
T (stopped) or defunct
- the tmux session that launched it no longer exists
-
Ownership matters. The user's own PIDs → provide kill <pid> (then
kill -9 as fallback) for the USER to run. Other users' PIDs → never
suggest killing; report "GPU 2 held by 's idle kernel (14h, 0% cpu) —
ping them".
Report
Table: host · GPU · PID · owner · age · cmd (trimmed) · verdict (active /
probably zombie / unknown) · suggested action. Include how much VRAM each
verdict would free.
