Use this skill when managing Linux servers, writing shell scripts, configuring systemd services, debugging networking, or hardening security. Triggers on bash scripting, systemd units, iptables, firewall, SSH configuration, file permissions, process management, cron jobs, disk management, and any task requiring Linux system administration.
Use this skill when managing Linux servers, writing shell scripts, configuring systemd services, debugging networking, or hardening security. Triggers on bash scripting, systemd units, iptables, firewall, SSH configuration, file permissions, process management, cron jobs, disk management, and any task requiring Linux system administration.
When this skill is activated, always start your first response with the 🧢 emoji.
Linux Administration
A production-focused Linux administration skill covering shell scripting, service
management, networking, and security hardening. This skill treats every Linux system
as a production asset - configuration is explicit, changes are auditable, and security
is a constraint from the start, not an afterthought. Designed for engineers who need
to move confidently between writing a deploy script, debugging a network issue, and
locking down a fresh server.
When to use this skill
Trigger this skill when the user:
Writes or debugs a bash script (especially anything running in CI, cron, or production)
Creates or modifies a systemd service, timer, socket, or target unit
Configures or audits SSH daemon settings and access controls
Debugs a networking issue (routing, DNS, firewall, port connectivity)
Sets up or modifies iptables/nftables/ufw firewall rules
Manages file permissions, ownership, ACLs, or setuid/setgid bits
Monitors or investigates running processes (CPU, memory, open files, syscalls)
Sets up cron jobs or scheduled tasks
Manages disk space, log rotation, or filesystem mounts
Do NOT trigger this skill for:
Container orchestration specifics (Kubernetes networking, Docker Compose config) - use a
Docker/K8s skill instead
Cloud provider IAM, VPC routing, or managed service configuration - those are cloud
platform concerns, not OS-level Linux administration
Key principles
Principle of least privilege - Every process, user, and service should run with
the minimum permissions required. Use dedicated service accounts (not root), restrict
file permissions to exactly what is needed, and audit sudo rules regularly.
Automate repeatable tasks - If you run a command twice, script it. Scripts should
be idempotent - running them again should produce the same result, not break things.
Store scripts in version control.
Log everything that matters - Structured logs, audit logs (auditd), and systemd
journal entries are your incident response safety net. Log authentication events,
privilege escalations, and configuration changes. Log rotation prevents disk exhaustion.
Immutable servers when possible - Prefer rebuilding servers from a known-good
image over patching in place. Use configuration management (Ansible, cloud-init) to
define state declaratively. Manual "snowflake" servers drift and fail unpredictably.
- Every script, service unit, and firewall rule change should be
validated in a non-production environment first. Use , , and
to validate before applying.
Test in staging
--dry-run
bash -n
iptables --check
Core concepts
File permissions
Linux permissions have three layers (owner, group, others) and three bits (read, write,
execute). Octal notation is the authoritative form.
Octal Symbolic Meaning
0 --- no permissions
1 --x execute only
2 -w- write only
4 r-- read only
6 rw- read + write
7 rwx read + write + execute
# Common patterns
chmod 600 ~/.ssh/id_rsa # private key: owner read/write only
chmod 644 /etc/nginx/nginx.conf # config: owner rw, others read
chmod 755 /usr/local/bin/script # executable: owner rwx, others rx
chmod 700 /root/.gnupg # directory: only owner can enter
Special bits:
setuid (4xxx): executable runs as file owner, not caller. Dangerous on scripts.
setgid (2xxx): new files in directory inherit group. Useful for shared dirs.
sticky (1xxx): only file owner can delete in a directory (e.g., /tmp).
Process management
Key signals for process control:
Signal
Number
Meaning
SIGTERM
15
Polite shutdown - process should clean up
SIGKILL
9
Immediate kill - kernel enforced, unblockable
SIGHUP
1
Reload config (many daemons re-read on SIGHUP)
SIGINT
2
Interrupt (Ctrl+C)
SIGUSR1/2
10/12
Application-defined
niceness runs from -20 (highest priority) to 19 (lowest). Use nice -n 10 cmd for
background tasks and renice to adjust running processes.
Dependency directives: Requires= (hard), Wants= (soft), After= (ordering only).
After=network-online.target is the correct way to wait for network connectivity.
Networking stack
Key tools and their roles:
Tool
Layer
Purpose
ip addr / ip link
L2/L3
Interface state, IP addresses, routes
ip route
L3
Routing table inspection and management
ss -tulpn
L4
Listening ports, socket state, owning process
iptables -L -n -v
L3/L4
Firewall rules, packet counts
dig / resolvectl
DNS
Name resolution debugging
traceroute / mtr
L3
Path tracing, hop-by-hop latency
tcpdump
L2-L7
Packet capture for deep inspection
Common tasks
Write a robust bash script
Always use the safety triplet at the top of every non-trivial script.
#!/usr/bin/env bashset -euo pipefail
# -e: exit on error# -u: treat unset variables as errors# -o pipefail: pipeline fails if any command in it fails# Cleanup on exit - runs on success, error, and signals
TMPDIR_WORK=""cleanup() {
local exit_code=$?
[[ -n "$TMPDIR_WORK" ]] && rm -rf "$TMPDIR_WORK"exit"$exit_code"
}
trap cleanup EXIT INT TERM
# Argument parsing with defaults and validationusage() {
echo"Usage: $0 [-e ENV] [-d] <target>"echo" -e ENV Environment (default: staging)"echo" -d Dry-run mode"exit 1
}
ENV="staging"
DRY_RUN=falsewhilegetopts":e:dh" opt; docase$optin
e) ENV="$OPTARG" ;;
d) DRY_RUN=true ;;
h) usage ;;
:) echo"Option -$OPTARG requires an argument." >&2; usage ;;
\?) echo"Unknown option: -$OPTARG" >&2; usage ;;
esacdoneshift $((OPTIND - 1))
[[ $# -lt 1 ]] && { echo"Error: target required" >&2; usage; }
TARGET="$1"# Use mktemp for safe temp directories
TMPDIR_WORK=$(mktemp -d)
# Log with timestampslog() { echo"[$(date '+%Y-%m-%dT%H:%M:%S')] $*"; }
log"Starting deploy: env=$ENV target=$TARGET dry_run=$DRY_RUN"# Dry-run wrapperrun() {
if [[ "$DRY_RUN" == true ]]; thenecho"[DRY-RUN] $*"else"$@"fi
}
run rsync -av --exclude='.git'"./""deploy@${TARGET}:/opt/app/"log"Deploy complete"
Create a systemd service unit
A service + timer pair for a scheduled task (replacing cron):
# /etc/systemd/system/db-backup.service[Unit]Description=Database backup
After=network-online.target postgresql.service
Wants=network-online.target
# Prevent starting if PostgreSQL is not runningRequires=postgresql.service
[Service]Type=oneshot
User=backup
Group=backup
# Security hardeningNoNewPrivileges=trueProtectSystem=strict
ProtectHome=trueReadWritePaths=/var/backups/db
PrivateTmp=trueExecStart=/usr/local/bin/db-backup.sh
StandardOutput=journal
StandardError=journal
# Retry on failureRestart=on-failure
RestartSec=60[Install]WantedBy=multi-user.target
# /etc/systemd/system/db-backup.timer[Unit]Description=Run database backup daily at 02:00Requires=db-backup.service
[Timer]# Run at 02:00 every dayOnCalendar=*-*-* 02:00:00# Run immediately if last run was missed (e.g., server was down)Persistent=true# Randomize start within 5 minutes to avoid thundering herdRandomizedDelaySec=300[Install]WantedBy=timers.target
# /etc/ssh/sshd_config - production hardening
# Use SSH protocol 2 only (default in modern OpenSSH, make it explicit)
Protocol 2
# Disable root login - use a dedicated admin user with sudo
PermitRootLogin no
# Disable password authentication - key-based only
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
# Disable X11 forwarding unless needed
X11Forwarding no
# Limit login window to prevent slowloris-style attacks
LoginGraceTime 30
MaxAuthTries 4
MaxSessions 10
# Only allow specific groups to SSH
AllowGroups sshusers admins
# Restrict ciphers, MACs, and key exchange to modern algorithms
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org
# Use privilege separation
UsePrivilegeSeparation sandbox
# Log at verbose level to capture key fingerprints on auth
LogLevel VERBOSE
# Set idle timeout: disconnect after 15 minutes of inactivity
ClientAliveInterval 300
ClientAliveCountMax 3
# Validate before restartingsudo sshd -t
# Restart sshd (keep current session open until verified)sudo systemctl restart sshd
# Verify from a NEW session before closing the old one
ssh -v user@host
Never close your existing SSH session until you have verified a new session works.
A broken sshd config can lock you out of the server permanently.
Debug networking issues
For detailed networking debugging workflow and firewall configuration (ufw and iptables), see references/networking-and-firewall.md.
Manage disk space
# Check disk usage overviewdf -hT
# -h: human readable -T: show filesystem type# Find large directories (top 10, depth-limited)du -h --max-depth=2 /var | sort -rh | head -10
# Interactive disk usage explorer (install ncdu first)
ncdu /var/log
# Find large files
find /var -type f -size +100M -execls -lh {} \; 2>/dev/null | sort -k5 -rh
# Check journal size and truncate if needed
journalctl --disk-usage
sudo journalctl --vacuum-size=500M # keep last 500MBsudo journalctl --vacuum-time=30d # keep last 30 days
# Test logrotate config without running it
logrotate --debug /etc/logrotate.d/myapp
# Force a rotation run
logrotate --force /etc/logrotate.d/myapp
Monitor processes
# Overview: CPU, memory, load average
top -b -n 1 -o %CPU | head -20 # batch mode, sort by CPU
htop # interactive, colored, tree view# Find what a process is doing
pid=$(pgrep -x nginx | head -1)
# Open files and network connections
lsof -p "$pid"# all open files
lsof -p "$pid" -i # only network connections
lsof -i :8080 # what process owns port 8080# System calls (strace) - use when a process behaves unexpectedly
strace -p "$pid" -f -e trace=network # network syscalls only
strace -p "$pid" -f -c # count syscall frequency (summary)
strace -c cmd arg # profile syscalls of a new command# Memory inspectioncat /proc/"$pid"/status | grep -E 'Vm|Threads'cat /proc/"$pid"/smaps_rollup # detailed memory breakdown# Check zombie/defunct processes
ps aux | awk '$8 == "Z" {print}'# Kill process tree (all children too)kill -TERM -"$(ps -o pgid= -p "$pid" | tr -d ' ')"
Error handling
Error
Likely cause
Resolution
Permission denied (publickey) on SSH
Wrong key, wrong user, or sshd config restricts access
Check ~/.ssh/authorized_keys permissions (must be 600), verify AllowGroups in sshd_config, run ssh -v for detail
Unit not found in systemctl
Unit file not in a searched path or daemon not reloaded
Run systemctl daemon-reload, verify unit file path with systemctl show -p FragmentPath
Job for X failed. See journalctl -xe
Service exited non-zero at startup
Run journalctl -u service-name -n 50 --no-pager to see startup errors
RTNETLINK answers: File exists when adding route
Route already exists in the routing table
Check with ip route show, delete conflicting route with ip route del, then re-add
iptables: No chain/target/match by that name
Missing kernel module or typo in chain name
Load module with modprobe xt_conntrack, check spelling of built-in chains (INPUT, OUTPUT, FORWARD)
Script exits unexpectedly with no error message
set -e triggered on a command that returned non-zero
Add `
Gotchas
set -e silently swallows exit codes in conditionals - if cmd; then or cmd || true suppress the exit code and bypass set -e. This is expected behavior but surprises people when a critical command fails without aborting the script. Use explicit exit code checks (rc=$?; if [[ $rc -ne 0 ]]; then) when a failure must be detected inside a conditional.
Restarting sshd locks you out if config is invalid - Always run sshd -t to validate config before restarting. Then restart sshd and verify from a new terminal session before closing the old one. A broken sshd_config or missing authorized_keys file after a restart leaves the server completely inaccessible.
iptables rules are not persistent across reboots by default - Rules applied via iptables commands are in-memory only. On reboot, they vanish. Use iptables-save > /etc/iptables/rules.v4 and install iptables-persistent, or use ufw which handles persistence automatically.
systemd After= is ordering-only, not a dependency - After=network.target does not guarantee the network is actually up; it only means the service starts after that target is reached. Use After=network-online.target combined with Wants=network-online.target if the service genuinely needs a routed network connection at start.
du and df disagree when deleted files are held open - A process that deleted a large log file but still has an open file descriptor causes df to show the disk as full while du shows free space. Find the culprit with lsof +L1 (lists open files with zero link count) and restart or signal the process to release the handle.
References
For detailed guidance on specific security domains, read the relevant file from
the references/ folder:
references/security-hardening.md - SSH, firewall, user management, kernel
hardening params, and audit logging checklist
Only load the references file when the current task requires it - it is detailed and
will consume context.
Companion check
On first activation of this skill in a conversation: check which companion skills are installed by running ls ~/.claude/skills/ ~/.agent/skills/ ~/.agents/skills/ .claude/skills/ .agent/skills/ .agents/skills/ 2>/dev/null. Compare the results against the recommended_skills field in this file's frontmatter. For any that are missing, mention them once and offer to install:
