disk-cleanup-codex-klava

name	disk-cleanup-codex-klava
description	Free space on the codex-klava root disk by purging /tmp junk and moving large data dirs to the mounted volume with symlinks. Use when root disk > 85% or Vadim says "free space", "disk full", "move to mnt".
user_invocable	true
tags	["devops","disk","codex-klava"]

Disk Cleanup — codex-klava

Codex-klava has a 38 GB root (/dev/sda1) and a 98 GB volume (/dev/sdb → /mnt/HC_Volume_106076117). Root fills easily; the volume stays at ~30–40%.

Trigger

Run when:

• df -h / shows ≥ 85% used
• Vadim says "free space", "disk full", "move to mnt"
• A build/download is about to fail from ENOSPC

Permission constraints

codex is NOT in sudoers. Two consequences:

1. Cannot create new top-level dirs on the volume root — mkdir /mnt/HC_Volume_106076117/newdir → Permission denied.
2. Cannot remove root-owned files/dirs (stale Codex lock dirs, git objects written by root, etc.). These get left behind after a partial mv.
3. Cannot journalctl --vacuum, apt clean, or truncate /var/log rotated logs.

Workaround for (1): Write inside a codex-owned subdir — use /mnt/HC_Volume_106076117/IP_Evidence/home-codex/ as the staging area. It is owned by codex.

Step-by-step

1. Assess

df -h / /mnt/HC_Volume_106076117
# Then find the biggest consumers:
du -sh /srv/codex-klava/data/* 2>/dev/null | sort -rh | head -20
du -sh /home/codex/.* 2>/dev/null | sort -rh | head -20
du -sh /tmp/* 2>/dev/null | sort -rh | head -20
du -sh /srv/codex-klava/venvs/* 2>/dev/null | sort -rh

2. Nuke /tmp first (free space fast, no data loss)

Safe to delete unconditionally:

• venvs: *-venv, *_venv, dirs ending in -venv
• bundles: *.bundle, *.tgz (caterpillar backups, etc.)
• old sqlite dumps, result dirs, old site builds

rm -rf /tmp/*-venv /tmp/*_venv /tmp/*.bundle /tmp/*.tgz \
       /tmp/wandb-* /tmp/caterpillar-* /tmp/obgyn-* /tmp/airlab-*
find /tmp -maxdepth 1 -name "tmp.*" -empty -delete 2>/dev/null
find /tmp -maxdepth 1 \( -name "*.err" -o -name "*.log" \) -empty -delete 2>/dev/null

3. Move large dirs to volume + symlink

Destination base: /mnt/HC_Volume_106076117/IP_Evidence/home-codex/

Pattern for each dir:

MNT=/mnt/HC_Volume_106076117/IP_Evidence/home-codex
mv /path/to/dir $MNT/dest-name && ln -s $MNT/dest-name /path/to/dir && echo "✓"

Priority order (biggest first, safest first):

Dir	Typical size	Notes
data/vadimgest/markdown	2.7 GB	vadimgest search still works via symlink
data/vadimgest/sources	1.3 GB	same
data/MyBrain	2.8 GB	Obsidian vault; works fine via symlink
data/klava	784 MB	daily memory, runtime state
repos/claude	1.5 GB	main klava repo; see pitfall below
~/.codex	2.6 GB	Headroom/Codex data; see pitfall below
~/.vscode-server	1.2 GB	safe to move
~/.npm + ~/.npm-global	1.3 GB	safe
~/.cache	429 MB	safe
venvs/headroom	625 MB	rarely used; klava venv stays on root
data/hermes/lsp	88 MB	static binaries
data/hermes/bin	80 MB	static executables
data/hermes/cron/output	62 MB	old cron run output
~/Documents/GitHub	246 MB	safe

4. Handle the state.db (hermes)

data/hermes/state.db is 550 MB and actively used by the live gateway. Do NOT move it — moving it requires a gateway restart and risks corruption. Leave it on root.

5. Items that need sudo (tell Vadim)

sudo apt clean                             # clears /var/cache/apt/archives (~150 MB)
sudo journalctl --vacuum-size=200M         # trims systemd journal (~1 GB)
sudo rm -rf /var/log/syslog.1 /var/log/auth.log.1 /var/log/kern.log.1 /var/log/ufw.log.1

Pitfalls

mv times out on large dirs (> ~1 GB) crossing filesystems

mv across mount points is a copy+delete, not a rename. 2.7 GB takes > 3 min and will hit the 180s terminal timeout.

Fix: Let it timeout — the copy usually completes. Then:

# Check if dest is full:
du -sh $MNT/dest-name   # should be full size
# Remove source and symlink:
rm -rf /original/path && ln -s $MNT/dest-name /original/path

Partial mv leaves source with root-owned files

When mv hits root-owned files mid-operation, it stops. The source dir still exists with whatever didn't move.

Fix:

# Sync remainder with rsync (skip errors from root-owned files)
rsync -a --ignore-errors /source/ $MNT/dest/
# Remove codex-owned files from source
find /source -not -path "*/tmp/*" -type f -delete 2>/dev/null
# Remove empty dirs (skip root-owned ones)
find /source -not -path "*/tmp/*" -mindepth 1 -maxdepth 3 -type d -empty -delete 2>/dev/null

The remaining ~80KB of root-owned lock files in .codex/tmp/arg0/ are safe to leave; they're stale symlinks pointing at /usr/local/lib/node_modules/....

.codex can't be fully symlinked while root-owned tmp exists

The .codex/tmp/arg0/codex-arg0*/ dir is created by the Codex CLI running as root. It leaves stale files codex can't remove.

Workaround: Move the bulk of .codex to mnt, do rsync --ignore-errors for the remainder, remove codex-owned files, and leave the root-owned tmp stub in place. The remaining dir size is <1 MB.

repos/claude has root-owned git objects

Some .git/objects/ files were written by root during previous codex runs. The mv will partially fail.

Fix: The mnt copy will be complete (mv copies before deleting). Then:

rm -rf /srv/codex-klava/repos/claude   # removes what it can
# A `claude-root-leftovers` dir stays with ~7.9 MB of root-owned files
mv /srv/codex-klava/repos/claude /srv/codex-klava/repos/claude-root-leftovers
ln -s $MNT/repos-claude /srv/codex-klava/repos/claude

⚠️ Moving ~/.local/share/uv/ breaks the Hermes venv python

The Hermes venv at /srv/codex-klava/apps/hermes-agent/venv/bin/ uses symlinks pointing at the uv-managed Python 3.11 binary:

venv/bin/python → /home/codex/.local/share/uv/python/cpython-3.11-linux-x86_64-gnu/bin/python3.11

If ~/.local/share/uv/ gets moved to the volume (e.g. as part of .local cleanup), the binary path changes and all no_agent cron scripts fail with:

Script execution failed: [Errno 2] No such file or directory: '/srv/codex-klava/apps/hermes-agent/venv/bin/python'

Symptoms: 3+ cron jobs flip to error after a disk cleanup session. The running Hermes process stays alive (it loaded into memory before the move), but new script invocations fail.

Fix: Find the new Python 3.11 binary location and update all three venv symlinks:

# Find the moved Python binary (version may differ — check cpython-3.11.15-* dirs)
NEW_PYTHON=$(find /mnt/HC_Volume_106076117 -name "python3.11" -type f 2>/dev/null | head -1)
echo "Found: $NEW_PYTHON"

# Update the three venv symlinks
cd /srv/codex-klava/apps/hermes-agent/venv/bin/
ln -sf "$NEW_PYTHON" python
ln -sf "$NEW_PYTHON" python3
ln -sf "$NEW_PYTHON" python3.11

# Verify
./python3 --version   # should print Python 3.11.x
hermes version        # should show Hermes Agent vX with Python 3.11.x

Prevention: When moving ~/.local/ or any of its subdirs, explicitly skip ~/.local/share/uv/python/ — add it to the exclusion list. Alternatively, move it and immediately update the venv symlinks as above before any cron jobs fire.

Note on execute_code tool: After this break, execute_code will also fail with the same error — it runs through the same broken venv. Use terminal() directly for Python work until the symlinks are restored.

Verification

df -h / /mnt/HC_Volume_106076117
# Confirm key symlinks work:
ls /srv/codex-klava/data/MyBrain | head -3
ls /srv/codex-klava/data/vadimgest/markdown | head -3
ls /home/codex/.vscode-server | head -3
# Check no hermes tools are broken:
VADIMGEST_CONFIG=/srv/codex-klava/data/vadimgest/config.yaml \
  /srv/codex-klava/venvs/klava/bin/python -m vadimgest stats 2>/dev/null | head -5

Expected result

State	Root usage
Before	~98% (35 GB used)
After /tmp purge	~88% (33 GB)
After all moves	~45–50% (17–19 GB)

Volume goes from ~20% → ~35% used (plenty of headroom).