| name | gpu-node-orphans |
| description | Use when GPU pods or PVCs linger with no owner reference and accelerator spend keeps climbing, when a finished training run left volumes behind, or when someone asks to "reclaim idle GPUs" or "clean up orphaned GPU nodes." Finds true orphans (no owner ref AND no recent activity), posts them to Slack for a soak period, and only after explicit human confirmation performs a guarded, cascading cleanup. Pushes back on deleting anything merely idle. |
GPU Node Orphan Reclamation
Overview
GPU capacity is the most expensive resource in the fleet, and the easiest to leak.
A training job crashes, its pod loses its owning Job, the PVC keeps its disk, and
the accelerator sits reserved but unused — quietly billing. This skill finds those
genuinely orphaned GPU pods and PersistentVolumeClaims, broadcasts the list for
human review, waits out a soak period, and then performs a guarded cascading
cleanup that cannot fire without an explicit confirmation token.
The hard rule: idle is not orphaned. A resource qualifies only when it has no
owner reference AND has shown no activity within the soak window. Both checks, every
time.
When to Use
Use this skill when you see symptoms like:
- GPU utilization dashboards show reserved-but-0%-busy accelerators for hours/days.
kubectl get pods shows GPU pods with no ownerReferences and no recent restarts.- Accelerator spend is climbing with no matching increase in active training runs.
- Old PVCs remain after their jobs completed, holding disks (and cost).
- Someone asks to "reclaim idle GPUs," "find orphaned GPU nodes," or "clean up after that failed run."
Do NOT use this skill when:
- A long-running job is merely idle between steps — it may be owned and wanted.
- You cannot confirm a soak period has elapsed (do not shortcut the wait).
- The user has not yet seen and approved the candidate list (never cascade blind).
