zombie-hunt

name	zombie-hunt
description	Find zombie processes holding GPU VRAM on lab servers (dead notebook kernels, orphaned trainers) and propose kill commands. Report-only. Use when GPUs look occupied but idle, or for "who is using GPU N".

Zombie hunt (report-only)

Goal: explain every GPU whose memory is held, and flag holders that are probably dead weight. You never send a signal yourself.

1. list_hosts → hosts where GPUs are inUse. For each, via run_command:

2. Map memory to processes:

   nvidia-smi --query-compute-apps=gpu_uuid,pid,process_name,used_memory --format=csv,noheader
   nvidia-smi --query-gpu=index,uuid --format=csv,noheader
   

3. Inspect each PID:

   ps -o pid,ppid,user,etime,stat,%cpu,cmd -p <pid1>,<pid2>,...
   

4. Zombie heuristics — call it a zombie only on 2+ signals (or 1 signal plus user confirmation):

   • jupyter/ipykernel holding VRAM with ~0 %cpu over a long etime
   • orphaned trainer (ppid 1) with 0% GPU util across two samples ~15s apart
   • process state T (stopped) or defunct
   • the tmux session that launched it no longer exists

5. Ownership matters. The user's own PIDs → provide kill <pid> (then kill -9 as fallback) for the USER to run. Other users' PIDs → never suggest killing; report "GPU 2 held by 's idle kernel (14h, 0% cpu) — ping them".

Report

Table: host · GPU · PID · owner · age · cmd (trimmed) · verdict (active / probably zombie / unknown) · suggested action. Include how much VRAM each verdict would free.