메뉴
Use on any NixOS host (devbox on Hetzner, cloudbox on GCP) when SSH connection fails, host key mismatch, NixOS issues, CPU/IO contention (high load), or verifying the host is properly configured
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
Launch headless opencode sessions from CLI. Use when you need to start a new opencode session in the background to work on a task in parallel, or when spawning work on a specific directory.
How to apply configuration changes to NixOS hosts (devbox, cloudbox). Use when you need to rebuild the system, apply home-manager changes, or recover from issues.
Use when setting up or rotating PagerDuty MCP auth for OpenCode on macOS Keychain or cloudbox sops, or when debugging missing PagerDuty tools.
Use when setting up or rotating Rollbar MCP auth for OpenCode on macOS Keychain or cloudbox sops, when triaging a Rollbar error paged via PagerDuty, or when debugging missing Rollbar tools.
Documents the OpenCode agent set — what each does, when to use it, and why the others were cut. Use when questioning agent choices or considering adding/removing agents.
Use when operating, deploying, debugging, or extending the aigateway LLM cost-capture proxy on cloudbox — routing opencode's Vertex providers through it, querying the per-request cost ledger (gateway_request_log), adding model prices, or rolling routing back.
| name | troubleshooting-nixos-host |
| description | Use on any NixOS host (devbox on Hetzner, cloudbox on GCP) when SSH connection fails, host key mismatch, NixOS issues, CPU/IO contention (high load), or verifying the host is properly configured |
Common issues and fixes for the NixOS remote development environments — devbox
(Hetzner) and cloudbox (GCP ARM). Most content is generic to both hosts;
sections that are truly host-specific (Hetzner rescue mode, hcloud commands)
are marked (devbox-only) inline.
Interactive sessions use mosh for persistence (survives sleep/wake, network changes):
mosh devbox -- tmux attach -t 1 # Persistent session via mosh + tmux
mosh cloudbox -- tmux attach -t 1
Port forwarding runs via persistent SSH tunnel launchd agents on macOS (always-on, independent of interactive sessions):
| Agent | SSH Host | Purpose |
|---|---|---|
devbox-dev-tunnel | devbox-tunnel | Dev ports + gclpr + CDP + chatgpt-relay |
cloudbox-dev-tunnel | cloudbox-tunnel | Dev ports + gclpr + CDP + chatgpt-relay |
Check tunnel status:
launchctl list | grep dev-tunnel
Restart a tunnel:
launchctl kickstart -k gui/$(id -u)/org.nix-community.home.devbox-dev-tunnel
Ports are configured in scripts/update-ssh-config.sh and carried by the persistent tunnel agents.
| Port | Direction | Purpose |
|---|---|---|
| 4000 | Local (-L) | eternal-machinery dev server |
| 4003 | Local (-L) | eternal-machinery (secondary) |
| 4005 | Local (-L) | eternal-machinery (tertiary) |
| 4173 | Local (-L) | citadels Vite dev server |
| 4400 | Local (-L) | boldco /preview UI harness (npm run dev:preview) |
| 1455 | Local (-L) | OpenCode OAuth callback |
| 9222 | Remote (-R) | Chrome DevTools Protocol - project A |
| 9223 | Remote (-R) | Chrome DevTools Protocol - project B |
| 3033 | Remote (-R) | chatgpt-relay (/ask-question CLI) |
| 2850 | Remote (-R) | gclpr clipboard (copy/paste to macOS) |
| Port | Direction | Purpose |
|---|---|---|
| 1455 | Local (-L) | OpenCode OAuth callback |
| 3334 | Local (-L) | mcp-remote OAuth (default Atlassian) |
| 3335 | Local (-L) | mcp-remote OAuth (alt Atlassian) |
| 9222 | Remote (-R) | Chrome DevTools Protocol - project A |
| 9223 | Remote (-R) | Chrome DevTools Protocol - project B |
| 3033 | Remote (-R) | chatgpt-relay (/ask-question CLI) |
| 2850 | Remote (-R) | gclpr clipboard (copy/paste to macOS) |
Local forwards (-L): devbox/cloudbox service → accessible on macOS localhost Remote forwards (-R): macOS service → accessible on remote localhost
Note: Both tunnels share LocalForward 1455. The agents use ExitOnForwardFailure=no so whichever starts first gets 1455; the other logs a warning but keeps all other forwards alive. OAuth is interactive and one-at-a-time, so this is fine.
Each project that uses human-in-the-loop visual QA needs its own Chrome instance on macOS with a unique --remote-debugging-port and --user-data-dir. Both CDP ports are reverse-forwarded over the same SSH session. See each project's VQA skill for the specific Chrome launch command and MCP server name (chrome-devtools-9222 or chrome-devtools-9223).
Host key changed because server was recreated with new IP or reinstalled.
# Remove old host key
ssh-keygen -R devbox
ssh-keygen -R $(hcloud server ip devbox)
# Reconnect with new key acceptance
ssh -o StrictHostKeyChecking=accept-new devbox
Check NixOS version (substitute devbox or cloudbox for <host>):
ssh <host> 'nixos-version'
# Expected: 24.11... (Vicuna)
Check tools are installed:
ssh <host> 'tmux -V && nvim --version | head -1 && mise --version'
View current system generation:
ssh <host> 'sudo nix-env --list-generations --profile /nix/var/nix/profiles/system'
Rollback to previous generation:
ssh <host> 'sudo nixos-rebuild switch --rollback'
Check system journal for errors:
ssh <host> 'journalctl -b --priority=err'
If deployment fails mid-way, the server may be in an inconsistent state.
Check if server is accessible:
# Try root (during nixos-anywhere)
ssh root@$(hcloud server ip devbox) 'uname -a'
# Try dev (after completion)
ssh dev@$(hcloud server ip devbox) 'uname -a'
If stuck, use Hetzner rescue mode:
hcloud server enable-rescue devbox --type linux64
hcloud server reboot devbox
# Then SSH as root with rescue password from console
Or just destroy and recreate:
/rebuild
# Should fail (root login disabled):
ssh root@$(hcloud server ip devbox) 'echo test'
# Should work:
ssh dev@$(hcloud server ip devbox) 'echo test'
Check latency (devbox via Hetzner):
ping -c 5 $(hcloud server ip devbox)
Helsinki datacenter latency from US East: ~100-150ms (acceptable for terminal work).
For cloudbox, use the GCP external IP (gcloud compute instances describe cloudbox --format='value(networkInterfaces[0].accessConfigs[0].natIP)').
Check Hetzner console:
hcloud server list
hcloud server describe devbox
Power cycle if needed:
hcloud server reboot devbox
ps, top)# Check load (reads /proc directly, works even under extreme load)
ssh devbox 'cat /proc/loadavg'
# Count runnable + blocked processes
# Format: load1 load5 load15 running/total lastpid
# "running" >> cores = oversubscribed
# Check memory (if this times out, problem is CPU not memory)
ssh devbox 'cat /proc/meminfo | head -5'
# Process count by name (lighter than ps/top)
ssh devbox 'for f in /proc/[0-9]*/comm; do cat "$f" 2>/dev/null; done | sort | uniq -c | sort -rn | head -20'
Resource controls prevent this from becoming unrecoverable:
| Layer | Setting | Devbox | Cloudbox |
|---|---|---|---|
| nix-daemon | CPUSchedulingPolicy | batch | batch |
| nix-daemon | IOSchedulingClass | idle | idle |
| nix-daemon | max-jobs | 8 (of 16) | 2 (of 4) |
| nix-daemon | cores | 2 | 2 |
| pull-workstation | Nice | 15 | 15 |
| pull-workstation | CPUQuota | 200% | 200% |
| pull-workstation | IOSchedulingClass | idle | idle |
| user-1000 slice | TasksMax | 2048 | 512 |
| sshd | CPUWeight | 200 | 200 |
These ensure nix builds and background updates yield to interactive sessions (mosh, tmux, opencode). SSH gets elevated CPU scheduling priority.
If the box is unresponsive despite mitigations:
# From macOS — reboot via cloud API (doesn't need SSH)
hcloud server reboot devbox # Hetzner devbox
# For cloudbox: use GCP console or gcloud CLI
Tmux sessions are lost but the load clears. Mosh will reconnect automatically after reboot.
After a reboot, check the previous boot's journal:
# Resource accounting per tmux session
journalctl -b -1 --no-pager -u "user@1000.service" | grep "Consumed"
# Check for OOM kills
journalctl -b -1 -k | grep -i "oom\|killed"
# nix-daemon IO during the incident
journalctl -b -1 -u nix-daemon | tail -5
NixOS can accumulate old generations. Clean up:
ssh <host> 'sudo nix-collect-garbage --delete-older-than 7d'
ssh <host> 'sudo nix-store --optimise'
Check disk usage:
ssh <host> 'df -h /'
Both NixOS hosts run several user-level timers for automation.
# List all user timers
systemctl --user list-timers
# Specific timers
systemctl --user status pull-workstation.timer
systemctl --user status home-manager-auto-expire.timer
# Auto-update pull logs
journalctl --user -u pull-workstation -n 50
# Generation cleanup logs
journalctl --user -u home-manager-auto-expire -n 50
If a timer isn't active after home-manager switch:
# Reload systemd
systemctl --user daemon-reload
# Enable and start timer
systemctl --user enable --now pull-workstation.timer
See Automated Updates for full details on the update pipeline.
Each host (macOS, devbox, cloudbox) has its own SSH signing key at
~/.ssh/id_ed25519_signing. Git is configured with gpg.format = ssh
and signs commits locally on whichever host you are using; no agent
forwarding is involved.
GitHub validates SSH-signed commits via each host's public key registered
under Settings → SSH and GPG keys as a
Signing Key. Local verification (git verify-commit, git log --show-signature) uses ~/.config/git/allowed_signers, deployed via
home-manager from assets/git/allowed_signers.
git commit fails with error: gpg failed to sign the datagit log --show-signature shows No principal matchedgit config --get gpg.format # expect: ssh
git config --get user.signingkey # expect: ~/.ssh/id_ed25519_signing.pub
git config --get gpg.ssh.allowedSignersFile # expect: ~/.config/git/allowed_signers
ls -la ~/.ssh/id_ed25519_signing* # both files present, private key 0600
ssh-keygen -l -f ~/.ssh/id_ed25519_signing.pub # fingerprint should match the
# corresponding line in
# ~/.config/git/allowed_signers
If gpg.format is missing or returns openpgp, home-manager has not
applied the SSH-signing config yet — re-run nix run home-manager -- switch --flake .#<host> (or darwin-rebuild on macOS).
ssh-keygen -t ed25519 -N '' -C '<email> (<host> signing)' -f ~/.ssh/id_ed25519_signingassets/git/allowed_signers with the format <email> namespaces="git" <pubkey> <hostname-tag>allowed_signers propagates: nix run home-manager -- switch --flake .#<host> (or darwin-rebuild on macOS)