validate-gpu-health

name	validate-gpu-health
description	GPU and VRAM health maintenance — validates CUDA availability, monitors VRAM usage, detects memory leaks, and ensures GPU readiness for training.

Validate GPU Health

Mental Model

GPU health is the foundation of ML training. When CUDA is unavailable, VRAM is leaked, or the GPU is otherwise unhealthy, training runs fail silently or crash mid-experiment. This skill ensures the GPU is ready for training, VRAM is properly managed, and memory leaks are detected before they waste GPU hours.

Coverage

Documented: CUDA availability, VRAM monitoring, memory leak detection, GPU readiness checks, cleanup verification. Not yet documented: Multi-GPU health, GPU temperature monitoring, driver version compatibility. Last extended: 2026-06-24

---

What This Skill Checks

1. CUDA Availability

Verification command:

python -c "
import torch
print(f'CUDA available: {torch.cuda.is_available()}')
print(f'CUDA version: {torch.version.cuda}')
print(f'GPU count: {torch.cuda.device_count()}')
if torch.cuda.is_available():
    print(f'GPU name: {torch.cuda.get_device_name(0)}')
    print(f'VRAM total: {torch.cuda.get_device_properties(0).total_mem / 1024**3:.1f} GB')
"

Checks:

• CUDA is available
• CUDA version matches PyTorch build
• At least one GPU is detected
• GPU name and VRAM are as expected (RTX 2080 Ti, 11 GB)

2. VRAM Monitoring

The project has notebooks/utils/vram_monitor.py for tracking VRAM during training.

Checks:

• VRAM monitor can be initialized
• VRAM monitor reports accurate readings
• VRAM usage is within expected bounds (GPT-2 small should use <4GB)

Verification command:

python -c "
import sys
sys.path.insert(0, 'notebooks/utils')
from vram_monitor import VRAMMonitor
monitor = VRAMMonitor()
monitor.start()
import torch
x = torch.randn(1000, 1000, device='cuda')
usage = monitor.get_usage()
print(f'VRAM usage: {usage:.2f} GB')
monitor.stop()
del x
torch.cuda.empty_cache()
print('VRAM monitor OK')
"

3. Memory Leak Detection

Training scripts should clean up GPU memory after use. Leaked memory accumulates across runs.

Checks:

• torch.cuda.empty_cache() is called after model deletion
• No leftover tensors on GPU after training
• VRAM returns to baseline after training completes

Verification command:

python -c "
import torch
before = torch.cuda.memory_allocated()
# Simulate training cleanup
torch.cuda.empty_cache()
after = torch.cuda.memory_allocated()
leaked = after - before
print(f'VRAM before: {before / 1024**2:.1f} MB')
print(f'VRAM after: {after / 1024**2:.1f} MB')
print(f'Leaked: {leaked / 1024**2:.1f} MB')
print('OK' if leaked < 1024**2 else 'LEAK DETECTED')
"

4. GPU Readiness for Training

Before running any notebook, verify the GPU is ready:

Checks:

• GPU is not in use by another process
• Sufficient VRAM is available (≥6GB free for GPT-2 small)
• GPU is not thermal throttling
• Driver version is compatible with CUDA version

Verification command:

nvidia-smi --query-gpu=memory.free,memory.total,utilization.gpu,temperature.gpu --format=csv

5. Training Cleanup Verification

After training completes, verify cleanup was performed:

Checks for experimental notebooks:

• del rig; torch.cuda.empty_cache() is called
• VRAM monitor is stopped (env["vram_monitor"].stop())
• No GPU tensors remain in memory

Checks for standard notebooks:

• run_standard_experiment() handles cleanup internally
• Verify no leaked memory after run

6. Mixed Precision Readiness

The project uses torch.cuda.amp for mixed precision training.

Checks:

• GPU supports FP16 (compute capability ≥ 7.0)
• torch.cuda.amp.autocast() works correctly
• GradScaler is available and functional

---

Anti-Patterns to Fix

1. CUDA unavailable — torch.cuda.is_available() returns False
2. VRAM leak — Memory not freed after training
3. Missing cleanup — No torch.cuda.empty_cache() call
4. VRAM monitor not stopped — Monitor thread continues after training
5. GPU occupied — Another process using the GPU
6. Insufficient VRAM — Not enough free memory for training
7. Driver mismatch — CUDA version doesn't match driver

---

Actionable Steps

1. Check CUDA Availability

python -c "import torch; print(f'CUDA: {torch.cuda.is_available()}, GPUs: {torch.cuda.device_count()})"

2. Check VRAM Status

nvidia-smi

3. Test VRAM Monitor

import sys
sys.path.insert(0, 'notebooks/utils')
from vram_monitor import VRAMMonitor
monitor = VRAMMonitor()
monitor.start()
# ... do something ...
monitor.stop()
print('VRAM monitor OK')

4. Check for Memory Leaks

Run a small test that allocates and frees GPU memory, then check if VRAM returns to baseline.

5. Verify Cleanup in Notebooks

Grep all notebooks for torch.cuda.empty_cache() and vram_monitor.stop().

6. Fix Issues

• Install CUDA toolkit if missing
• Kill processes using the GPU
• Add missing cleanup calls to notebooks
• Fix VRAM monitor initialization

---

Anti-Patterns to Fix (Specific)

Pattern	Where to Look	Fix
CUDA unavailable	torch.cuda.is_available()	Install CUDA toolkit, check driver
VRAM leak	torch.cuda.memory_allocated()	Add torch.cuda.empty_cache()
Missing cleanup	Notebooks	Add del rig; torch.cuda.empty_cache()
VRAM monitor running	End of notebook	Add env["vram_monitor"].stop()
GPU occupied	nvidia-smi	Kill other processes
Driver mismatch	nvidia-smi vs torch.version.cuda	Update driver or reinstall PyTorch

---

Known Violations

Check these specific locations first:

• notebooks/utils/vram_monitor.py — VRAM monitoring utility
• All experimental notebooks — may have missing cleanup
• notebooks/components/device_setup.py — Device initialization

---

Coverage

Already clean:

• CUDA availability (hardware dependent, not code issue)
• VRAM monitor utility exists and is functional

Still needs work:

• Cleanup verification in experimental notebooks
• Memory leak detection across training runs
• Mixed precision readiness checks

---

Verification

After fixing issues, verify:

# Check CUDA:
python -c "import torch; assert torch.cuda.is_available(), 'CUDA not available'"

# Check VRAM monitor:
python -c "
import sys
sys.path.insert(0, 'notebooks/utils')
from vram_monitor import VRAMMonitor
m = VRAMMonitor(); m.start(); m.stop()
print('VRAM monitor OK')
"

# Check cleanup in notebooks:
grep -l "torch.cuda.empty_cache" notebooks/*.py

# Check VRAM monitor stop:
grep -l "vram_monitor.*stop" notebooks/*.py

Report Format

STATUS: [no_work | fixed]

CHANGES:
- notebooks/01b_teacher_distillation.py: Added torch.cuda.empty_cache() after training
- notebooks/01c_aversion_sweep.py: Added vram_monitor.stop() call

DETAILS:
{Detailed explanation of each change}