cloud-server-maintenance

name	cloud-server-maintenance
category	devops
description	Assess, clean up, back up, and decide whether to keep or decommission a cloud VPS. Covers runaway process detection, disk cleanup, service evaluation, backup triage, and OS reinstallation workflow.

Cloud Server Maintenance

Use this skill when a user asks you to check on a cloud VPS, clean up disk space, decide whether to keep or decommission it, or prepare it for OS reinstallation.

Phase 1: Rapid Assessment

Connect and gather the essentials in one shot:

ssh root@<IP> "
  echo '=== OS ==='; cat /etc/os-release 2>/dev/null || cat /etc/redhat-release 2>/dev/null
  echo '=== RESOURCES ==='; free -h 2>/dev/null; df -h
  echo '=== UPTIME ==='; uptime
  echo '=== LISTENING PORTS ==='; netstat -tlnp 2>/dev/null || ss -tlnp
  echo '=== TOP PROCESSES ==='; ps aux --sort=-%mem | head -15
  echo '=== CRON ==='; crontab -l 2>/dev/null
  echo '=== INIT.D ==='; ls /etc/init.d/ 2>/dev/null | grep -v -E '^(functions|halt|killall|netfs|network|single|rc|netconsole|restorecond)$'
"

Phase 2: Identify Disk Hogs

Runaway daemon logs are the #1 cause of 100% disk on old VPS. Look for:

ssh root@<IP> "
  find /var/log -type f -size +50M -exec ls -lhS {} \; 2>/dev/null
  find /home -type f -size +100M -exec ls -lhS {} \; 2>/dev/null
  find /opt -type f -size +100M -name '*.tar.gz' -o -name '*.zip' 2>/dev/null
"

Common culprits:

• Daemon watchdog loops writing errors every few seconds (e.g., appdog, supervisor, custom scripts)
• Rotated logs that never get cleaned (access.log, secure, btmp, messages)
• Old package tarballs left in /opt/ or /root/ after manual installs

Fix: Stop the daemon first, THEN truncate

# 1. STOP the daemon (critical — truncate won't help if it keeps writing)
service <daemon> stop 2>/dev/null
kill -9 <PID> 2>/dev/null

# 2. Truncate logs (don't rm — truncating avoids "file not found" errors)
> /var/log/appdog.log
> /home/wwwlogs/access.log

# 3. Clean old rotated logs
rm -f /var/log/*-2017* /var/log/*.gz 2>/dev/null

# 4. Remove old package archives
rm -f /opt/gcc/*.tar.gz /opt/emq/*.zip 2>/dev/null

Verify: df -h should show significant space freed.

Phase 3: Service Evaluation Matrix

For each running service, evaluate:

Service	Memory	Keep?	Reason
sshd	~5MB	✅ Yes	Essential for access
nginx (stopped)	~5MB	⚠️ Depends	Only if serving sites
php-fpm (stopped)	~20MB	❌ Usually	Needs nginx too
MySQL (stopped)	~100MB	❌ On 512MB	Too heavy
Custom daemon (looping)	Variable	❌ No	Check if target still exists
cupsd	~5MB	❌ Usually	Print server, not needed

Low-memory VPS (≤512MB RAM) reality check

What it CAN do	Memory
SSH only (jump host)	~5MB
SSH + dnsmasq (DNS forwarder)	~7MB
SSH + cron scripts	~15MB
SSH + nginx static files	~10MB
What it CANNOT do
Jenkins	Needs 512MB+ Java heap alone
Docker	Kernel 2.6.32 doesn't support overlayfs
Node.js apps	V8 needs 100MB+ minimum
MySQL/PostgreSQL	100MB+ baseline
Tailscale Exit Node	Needs systemd + newer kernel

Phase 4: Backup Triage

Before OS reinstallation, systematically assess what's worth keeping:

Worth keeping (back up to local)

• Websites in /usr/share/ or /var/www/ — especially if domain still resolves here
• Database dumps or raw data directories — even .frm files + ibdata1 can be recovered
• SSL/TLS certificates and keys — self-signed CA, Let's Encrypt archives
• Custom configs in /etc/nginx/, /etc/init.d/ scripts
• SSH authorized_keys — record which keys were deployed

Can discard

• Package tarballs (.tar.gz, .zip) in /opt/ or /root/ — always re-downloadable
• Source code directories of open-source projects (nginx, ngrok, node.js) — available on GitHub
• Old logs (/var/log/*, /home/wwwlogs/*) — historical only
• .pyenv virtualenvs — easily recreated
• Build artifacts (nohup.out, .zcompdump, etc.)

Backup workflow

# On remote: tar up directories first (faster than scp -r for many small files)
ssh root@<IP> "cd /usr/share && tar czf /tmp/site_backup.tar.gz yudijiaoyu.com/"
ssh root@<IP> "cd /root && tar czf /tmp/db_backup.tar.gz databases_backup_*/"

# Pull compressed archives to local
mkdir -p ~/backup-<hostname>/{website,database,certs,configs}
scp root@<IP>:/tmp/site_backup.tar.gz ~/backup-<hostname>/website/
scp root@<IP>:/tmp/db_backup.tar.gz ~/backup-<hostname>/database/

# Clean up remote temp files
ssh root@<IP> "rm -f /tmp/*_backup.tar.gz"

Phase 5: DNS Resolution Check

If user reports "domains can't be opened":

# Check if DNS resolves to this IP
nslookup <domain> 2>&1 | grep "Address:"

# Check if nginx is listening
ss -tlnp | grep ':80'

# Check if nginx has vhost config for this domain
grep -r '<domain>' /etc/nginx/ 2>/dev/null

# Check HTTP response
curl -s -o /dev/null -w '%{http_code}' http://<domain>

Common pattern: DNS points to the IP, but:

1. Nginx is stopped → Connection refused
2. Nginx is running but no vhost config → 404 or default page
3. Nginx is running but upstream service is down → 502 Bad Gateway

Fix: Start nginx + add vhost config, OR update DNS to point to a different server.

Phase 6: OS Reinstallation Decision

Signs it's time to reinstall

• EOL OS (CentOS 6 EOL Nov 2020, CentOS 7 EOL June 2024)
• Disk corrupted or filesystem errors
• Can't install modern software (Docker, Node 18+, Python 3.10+)
• Too many accumulated configuration cruft

Before reinstalling

1. Complete Phase 4 (backup triage)
2. Record current DNS entries (screenshot or export)
3. Note the cloud provider's default login user (Ubuntu → ecs-user or ubuntu, not root)
4. Plan post-install steps: SSH key setup, firewall, zsh/omz, Tailscale

Phase 7: Post-Install Setup

After OS reinstallation, run these steps in order:

1. SSH key setup

The default user on Ubuntu ECS is ecs-user (not root). Root login may be disabled in sshd_config.

ssh ecs-user@<IP>
ssh-copy-id ecs-user@<IP>

Root key propagation: After OS reinstall, /root/.ssh/authorized_keys is wiped. ssh-copy-id root@<IP> fails non-interactively because it can't prompt for root's password. Workaround — pipe the key through ecs-user with sudo:

cat ~/.ssh/id_ed25519.pub | ssh ecs-user@<IP> \
  "sudo tee -a /root/.ssh/authorized_keys > /dev/null && sudo chmod 600 /root/.ssh/authorized_keys"

1b. Root zsh setup (optional)

After OS reinstall, root has no .zshrc and no oh-my-zsh. Instead of reinstalling everything for root, copy the working setup from ecs-user:

ssh root@<IP> '
  cp /home/ecs-user/.zshrc /root/.zshrc
  rm -rf /root/.oh-my-zsh
  cp -r /home/ecs-user/.oh-my-zsh /root/.oh-my-zsh
  chsh -s /usr/bin/zsh root
'

2. Install zsh + oh-my-zsh + plugins

ssh ecs-user@<IP> '
sudo apt update && sudo apt install -y zsh git curl zsh-autosuggestions zsh-syntax-highlighting

# Install oh-my-zsh (use mirror if GitHub times out)
sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)" "" --unattended

# Set as default shell
sudo chsh -s $(which zsh) ecs-user
'

Important: On Alibaba Cloud servers, GitHub git clone often times out. Install zsh plugins via apt install zsh-autosuggestions zsh-syntax-highlighting instead of cloning from GitHub. The apt packages install to /usr/share/zsh-*/ and are loaded by sourcing the .zsh files directly.

3. Write .zshrc with custom prompt

Use python3 heredoc to write .zshrc over SSH (avoids bash quoting hell). Use single quotes around PROMPT to prevent bash variable expansion:

ssh ecs-user@<IP> 'python3 << "PYEOF"
content = """export ZSH="$HOME/.oh-my-zsh"
ZSH_THEME="robbyrussell"
plugins=(git)
source $ZSH/oh-my-zsh.sh
# Load apt-installed plugins
source /usr/share/zsh-autosuggestions/zsh-autosuggestions.zsh
source /usr/share/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh
# Prompt: hostname + full path (%~ for full path, %m for hostname)
PROMPT='"'"'%(?:%{$fg_bold[green]%}➜ :%{$fg_bold[red]%}➜ ) %{$fg_bold[cyan]%}%m %{$fg[blue]%}%~%{$reset_color%} $(git_prompt_info)'"'"'
"""
with open("/home/ecs-user/.zshrc", "w") as f:
    f.write(content)
print("OK")
PYEOF'

4. Set hostname

ssh ecs-user@<IP> '
sudo hostnamectl set-hostname <new-hostname>
sudo sed -i "s/^127.0.1.1.*/127.0.1.1 <new-hostname>/" /etc/hosts
'

See references/post-install-zsh-setup.md for the complete prompt customization reference and verified .zshrc templates.

CentOS 6 specific: Tailscale cannot be installed

• No systemd (uses init.d)
• Kernel 2.6.32 too old for WireGuard userspace networking
• Alternative: Use ssh -D 1080 or autossh -M 0 -f -N -D 1080 <user>@<IP> for SOCKS5 proxy
  • Same effect as Exit Node for web browsing
  • Works on any system with sshd
  • Add to Mac login items for persistence
  • See references/tailscale-alternatives.md for detailed comparison and setup

Tailscale Exit Node Setup on Linux

When setting up a Linux VPS as a Tailscale Exit Node (for newer OS versions that support it):

Installation

curl -fsSL https://tailscale.com/install.sh | sh

IP Forwarding (required for Exit Node)

sysctl -w net.ipv4.ip_forward=1
sysctl -w net.ipv6.conf.all.forwarding=1
cat > /etc/sysctl.d/99-tailscale.conf << 'EOF'
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
EOF

Systemd Daemon Configuration

Critical: /etc/default/tailscaled is read by systemd. Only put daemon flags here:

cat > /etc/default/tailscaled << 'EOF'
PORT=41641
EOF

PITFALLS:

1. $FLAGS must NOT contain tailscale up flags (--accept-dns, --advertise-exit-node are client prefs, not daemon flags)
2. PORT must be set (empty string causes failure)
3. /etc/default/tailscaled (with d), NOT /etc/default/tailscale
4. After editing: systemctl daemon-reload && systemctl reset-failed tailscaled && systemctl restart tailscaled

Authenticating and Setting Exit Node

systemctl start tailscaled
tailscale up --accept-dns=false --accept-routes --advertise-exit-node

Architecture Summary

Layer	Config Location	What Goes Here
Daemon (systemd)	/etc/default/tailscaled	PORT=41641 only
Client prefs (state)	Stored via tailscale up	--accept-dns, --advertise-exit-node, etc.
Kernel	/etc/sysctl.d/	net.ipv4.ip_forward=1

UDP GRO Optimization

ethtool -K eth0 rx-gro-list on
ethtool -K eth0 rx-udp-gro-forwarding on

Troubleshooting

• status=2/INVALIDARGUMENT → invalid flag in $FLAGS, remove tailscale up flags
• invalid value "" for flag -port → PORT not set in /etc/default/tailscaled
• Start request repeated too quickly → systemctl reset-failed tailscaled

Pitfalls

• appdog/daemon loops: Always service stop + kill -9 BEFORE truncating logs. Truncating alone just creates a race where the daemon writes more data.
• scp -r on directories with many small files: Much slower than tar czf on remote then scp the archive. Always compress first.
• CentOS 6 yum failures: Repos are offline since EOL. yum install will fail. Use curl to download RPMs directly or use rpm -Uvh with archived packages.
• Ubuntu reinstall changes default user: After reinstalling to Ubuntu, the default user is ecs-user or ubuntu, NOT root. Root login may be disabled by default in sshd_config.
• Ubuntu 24.04 SSH port change via sshd_config doesn't work: Ubuntu 24.04 uses systemd socket activation (ssh.socket). Changing Port in /etc/ssh/sshd_config is silently ignored because ssh.socket controls the listening port. To change SSH port: systemctl disable --now ssh.socket && systemctl restart ssh.
• Cloud provider "security hardening" checkbox: On aliyun/腾讯云, the "free security hardening" (云安全中心) installs a daemon that uses 50-100MB RAM. On a 512MB VPS, this is 10-20% of total memory. Uncheck it during reinstall.