shell-scripting

When this skill is activated, always start your first response with the 🧢 emoji. # Shell Scripting Shell scripting is the art of automating tasks through the Unix shell - combining built-in commands, control flow, and process management to build reliable CLI tools and automation workflows. This skill covers production-quality bash and zsh scripting: robust error handling, portable argument parsing, safe file operations, and the idioms that separate fragile one-liners from scripts that hold up in production. --- ## When to use this skill Trigger this skill when the user: - Asks to write or review a bash or zsh script - Needs to parse command-line arguments or flags - Wants to automate a CLI workflow or task runner - Asks about exit codes, signal trapping, or error handling in shell - Needs to process files, lines, or streams from the terminal - Asks about here documents, process substitution, or subshells - Wants a portable script that works across bash, zsh, and sh Do NOT trigger this skill for: - Python or Node.js CLI tools (shell is the wrong tool for complex logic) - Scripts that require structured data parsing at scale (use a real language instead) --- ## Key principles 1. **Always use `set -euo pipefail`** - Start every non-trivial script with this. `-e` exits on error, `-u` treats unset variables as errors, `-o pipefail` catches failures in pipelines. Without this, silent failures hide bugs for weeks. 2. **Quote everything** - Always double-quote variable expansions: `"$var"`, `"$@"`, `"${array[@]}"`. Unquoted variables break on whitespace and glob characters. The only exceptions are intentional word splitting and arithmetic contexts. 3. **Check dependencies upfront** - Verify required commands exist before the script runs. Fail fast at the top with a clear error, not halfway through a destructive operation. 4. **Use functions for reuse and readability** - Extract logic into named functions. Shell functions support local variables (`local`), can return exit codes, and make scripts testable. A `main()` function at the bottom with a guard is idiomatic. 5. **Prefer shell built-ins over external commands** - `[[ ]]` over `[ ]`, `${var##*/}` over `basename`, `${#str}` over `wc -c`. Built-ins are faster, more portable, and avoid spawning subshells. Use `printf` over `echo` for reliable output formatting. --- ## Core concepts **Exit codes** - Every command returns an integer 0-255. `0` means success; any non-zero value means failure. Use `$?` to read the last exit code. Use explicit `exit N` to return meaningful codes from scripts. The `||` and `&&` operators branch on exit code. **File descriptors** - `0` = stdin, `1` = stdout, `2` = stderr. Redirect stderr with `2>file` or merge it into stdout with `2>&1`. Use `>&2` to write errors to stderr so they don't pollute captured output. **Subshells** - Parentheses `(cmd)` run commands in a child process. Changes to variables, `cd`, or `set` inside a subshell do not affect the parent. Command substitution `$(cmd)` also runs in a subshell and captures its stdout. **Variable scoping** - All variables are global by default. Use `local` inside functions to limit scope. `declare -r` creates read-only variables. `declare -a` declares arrays; `declare -A` declares associative arrays (bash 4+). **IFS (Internal Field Separator)** - Controls how bash splits words and lines. Default is space/tab/newline. When reading files line by line, set `IFS=` to prevent trimming of leading/trailing whitespace: `while IFS= read -r line`. --- ## Common tasks ### Robust script template with trap cleanup Every production script should start with this foundation: ```bash #!/usr/bin/env bash set -euo pipefail # --- constants --- readonly SCRIPT_NAME="$(basename "$0")" readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" readonly TMP_DIR="$(mktemp -d)" # --- cleanup --- cleanup() { local exit_code=$? rm -rf "$TMP_DIR" if [[ $exit_code -ne 0 ]]; then echo "ERROR: $SCRIPT_NAME failed with exit code $exit_code" >&2 fi exit "$exit_code" } trap cleanup EXIT INT TERM # --- dependency check --- require_cmd() { if ! command -v "$1" &>/dev/null; then echo "ERROR: required command '$1' not found" >&2 exit 1 fi } require_cmd curl require_cmd jq # --- main logic --- main() { echo "Running $SCRIPT_NAME from $SCRIPT_DIR" # ... your logic here } main "$@" ``` The `trap cleanup EXIT` fires on any exit - success, error, or signal - ensuring temp files are always removed. `BASH_SOURCE[0]` resolves the script's real location even when called via symlink. ### Argument parsing with getopts and long opts Use `getopts` for POSIX-portable short flags. For long options, use a `while/case` loop with manual shift: ```bash usage() { cat >&2 <<EOF Usage: $SCRIPT_NAME [OPTIONS] <input> Options: -o, --output <dir> Output directory (default: ./out) -v, --verbose Enable verbose logging -h, --help Show this help EOF exit "${1:-0}" } OUTPUT_DIR="./out" VERBOSE=false parse_args() { while [[ $# -gt 0 ]]; do case "$1" in -o|--output) [[ -n "${2-}" ]] || { echo "ERROR: --output requires a value" >&2; usage 1; } OUTPUT_DIR="$2"; shift 2 ;; -v|--verbose) VERBOSE=true; shift ;; -h|--help) usage 0 ;; --) shift; break ;; -*) echo "ERROR: unknown option '$1'" >&2; usage 1 ;; *) break ;; esac done # remaining positional args available as "$@" INPUT_FILE="${1-}" [[ -n "$INPUT_FILE" ]] || { echo "ERROR: input file required" >&2; usage 1; } } parse_args "$@" ``` ### File processing - read, write, and temp files safely ```bash # Read a file line by line without trimming whitespace or interpreting backslashes while IFS= read -r line; do echo "Processing: $line" done < "$input_file" # Read into an array mapfile -t lines < "$input_file" # bash 4+; equivalent: readarray -t lines # Write to a file atomically (avoids partial writes on failure) write_atomic() { local target="$1" local tmp tmp="$(mktemp "${target}.XXXXXX")" # write to tmp, then atomically rename cat > "$tmp" mv "$tmp" "$target" } echo "final content" | write_atomic "/etc/myapp/config" # Safe temp file with auto-cleanup (cleanup trap handles TMP_DIR removal) local tmpfile tmpfile="$(mktemp "$TMP_DIR/work.XXXXXX")" some_command > "$tmpfile" process_result "$tmpfile" ``` ### String manipulation without external tools ```bash # Substring extraction: ${var:offset:length} str="hello world" echo "${str:6:5}" # "world" # Pattern removal (greedy ##, non-greedy #; greedy %%, non-greedy %) path="/usr/local/bin/myapp" echo "${path##*/}" # "myapp" (strip longest prefix up to /) echo "${path%/*}" # "/usr/local/bin" (strip shortest suffix from /) # Search and replace filename="report-2024.csv" echo "${filename/csv/tsv}" # "report-2024.tsv" (first match) echo "${filename//a/A}" # "report-2024.csv" -> "report-2024.csv" (all matches) # Case conversion (bash 4+) lower="${str,,}" # all lowercase upper="${str^^}" # all uppercase title="${str^}" # capitalise first character # String length and emptiness checks [[ -z "$var" ]] && echo "empty" [[ -n "$var" ]] && echo "non-empty" echo "length: ${#str}" # Check if string starts/ends with a pattern (no grep needed) [[ "$str" == hello* ]] && echo "starts with hello" [[ "$str" == *world ]] && echo "ends with world" ``` ### Parallel execution with xargs and GNU parallel ```bash # xargs: run up to 4 jobs in parallel, one arg per job find . -name "*.log" -print0 \ | xargs -0 -P4 -I{} gzip "{}" # xargs with a shell function (must export it first) process_file() { local f="$1" echo "Processing $f" # ... work ... } export -f process_file find . -name "*.csv" -print0 \ | xargs -0 -P"$(nproc)" -I{} bash -c 'process_file "$@"' _ {} # GNU parallel (more features: progress, retry, result collection) # parallel --jobs 4 --bar gzip ::: *.log # parallel -j4 --results /tmp/out/ ./process.sh ::: file1 file2 file3 # Manual background jobs with wait pids=() for host in "${hosts[@]}"; do ssh "$host" uptime & pids+=($!) done for pid in "${pids[@]}"; do wait "$pid" || echo "WARN: job $pid failed" >&2 done ``` ### Portable scripts across bash, zsh, and sh ```bash # Detect the running shell detect_shell() { if [ -n "${BASH_VERSION-}" ]; then echo "bash $BASH_VERSION" elif [ -n "${ZSH_VERSION-}" ]; then echo "zsh $ZSH_VERSION" else echo "sh (POSIX)" fi } # POSIX-safe array alternative (use positional parameters) set -- alpha beta gamma for item do # equivalent to: for item in "$@" echo "$item" done # Use $(...) not backticks - both portable, but $() is nestable result=$(echo "$(date) - $(whoami)") # Avoid bashisms when targeting /bin/sh: # [[ ]] -> [ ] (but be careful with quoting) # local -> still works in most sh implementations (not POSIX but widely supported) # readonly var=val (POSIX-safe) # printf not echo -e (echo -e is not portable) printf '%s\n' "Safe output with no echo flag issues" ``` ### Interactive prompts and colored output ```bash # Color constants (no-op when not a terminal) setup_colors() { if [[ -t 1 ]]; then RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m' BLUE='\033[0;34m'; BOLD='\033[1m'; RESET='\033[0m' else RED=''; GREEN=''; YELLOW=''; BLUE=''; BOLD=''; RESET='' fi } setup_colors log_info() { printf "${GREEN}[INFO]${RESET} %s\n" "$*"; } log_warn() { printf "${YELLOW}[WARN]${RESET} %s\n" "$*" >&2; } log_error() { printf "${RED}[ERROR]${RESET} %s\n" "$*" >&2; } # Yes/no prompt confirm() { local prompt="${1:-Continue?} [y/N] " local reply read -r -p "$prompt" reply [[ "${reply,,}" == y || "${reply,,}" == yes ]] } # Prompt with default value prompt_with_default() { local prompt="$1" default="$2" value read -r -p "$prompt [$default]: " value echo "${value:-$default}" } # Spinner for long operations spin() { local pid=$1 msg="${2:-Working...}" local frames=('|' '/' '-' '\') local i=0 while kill -0 "$pid" 2>/dev/null; do printf "\r%s %s" "${frames[i++ % 4]}" "$msg" sleep 0.1 done printf "\r\033[K" # clear the spinner line } ``` --- ## Anti-patterns | Anti-pattern | Why it's wrong | What to do instead |