linux-service-integrity-operations

# Linux Service Integrity Operations Infrastructure engineer executing zero-downtime changes and deployments on Linux hosts using safe restart patterns, health monitoring during transitions, and automatic rollback procedures to maintain continuous service availability. ## TL;DR Checklist - [ ] Confirm the service supports graceful reload before considering restart - [ ] Set up a health check endpoint (HTTP /proc, or systemd-ready notification) before making changes - [ ] Verify rollback procedure and snapshot exist before starting the change - [ ] Execute the change with service-isolation (socket activation or port-based switching) - [ ] Monitor health for at least 60 seconds post-change with active health checks - [ ] Confirm dependent services have not entered degraded state - [ ] Document the change in runbook with before/after metrics --- ## When to Use Use this skill when: - **Deploying configuration changes to production services** — Applying new settings without unplanned downtime - **Performing kernel or library updates on running systems** — Services must remain available during package updates - **Upgrading application binaries or containers** — Zero-downtime deployment of new versions - **Modifying firewall rules or network configurations** — Maintaining active connections during policy changes - **Scaling services vertically** — Adjusting resource limits, CPU/memory quotas without service interruption - **Applying security patches to production workloads** — CVE fixes must not cause service outages --- ## When NOT to Use Avoid this skill for: - **Service-level failures where the service is already down** — Use incident response procedures instead - **Database migrations requiring schema changes** — Use dedicated database migration tools (pg_dump, flyway, etc.) - **Network-level re-architecting (VPC changes, subnet migrations)** — These require network planning and maintenance windows - **Complete host re-imaging or OS upgrades** — Requires planned downtime and rebuild procedures Use `linux-safe-script-execution` for pre-flight analysis before initiating zero-downtime operations. --- ## Core Workflow ### 1. Determine Service Restart Strategy Choose the correct restart strategy based on the service's capabilities and dependency requirements. ```bash #!/usr/bin/env bash # service_strategy.sh — Determine optimal change strategy for a systemd service # Usage: ./service_strategy.sh <service.service> set -euo pipefail determine_strategy() { local service="$1" if ! systemctl is-active --quiet "$service" 2>/dev/null; then echo "ERROR: Service '$service' is not running" >&2 return 1 fi echo "=== Change Strategy Analysis: ${service} ===" echo "" # Check if service supports reload local has_reload=false local exec_start exec_start=$(systemctl show "$service" --property=ExecStart --value 2>/dev/null || echo "") # Check unit file for ExecReload directive local unit_file unit_file=$(systemctl show "$service" --property=ExecReload --value 2>/dev/null || echo "") if [[ -n "$unit_file" && "$unit_file" != "-" ]]; then has_reload=true echo "Strategy: GRACEFUL RELOAD (preferred)" echo " ExecReload directive found: $unit_file" echo " Reload sends SIGHUP — connections maintained, no downtime" else echo " No ExecReload directive found — restart required" fi echo "" # Check service type for readiness signaling local svc_type svc_type=$(systemctl show "$service" --property=Type --value 2>/dev/null || echo "simple") echo "Service type: $svc_type" case "$svc_type" in notify) echo " Uses sd_notify() — systemd tracks readiness accurately" echo " Strategy: Signal READY after config load completes" ;; forking) echo " Forking process — systemd waits for parent exit" echo " Strategy: Fork handles config reload, parent exits" ;; oneshot) echo " One-shot service — not a long-running daemon" echo " Strategy: Re-run the oneshot with new parameters" ;; simple|exec|dbus|idle) echo " Simple/exec service — process is the main process" echo " Strategy: Send signal to main process directly" ;; esac echo "" # Check for socket activation local socket_unit socket_unit=$(systemctl list-dependencies "$service" --reverse --no-pager 2>/dev/null | grep '\.socket' | head -5 || true) if [[ -n "$socket_unit" ]]; then echo "Socket activation detected:" echo "$socket_unit" | sed 's/^/ /' echo " Benefit: systemd holds listening sockets during restart" echo " New connections queue until service is ready" else echo " No socket activation — brief connection gap during restart" fi echo "" # Determine health check method echo "=== Health Check Methods ===" if grep -q "ExecStartPre=" "$unit_file" 2>/dev/null || grep -q "ExecStart=" "$unit_file" 2>/dev/null; then local health_url health_url=$(grep -iE "(health|ready|status|ping)" "$unit_file" 2>/dev/null | grep -iE "exec|cmd" || true) if [[ -n "$health_url" ]]; then echo " Built-in health endpoint found" else echo " No explicit health endpoint — use systemctl is-active" fi fi # Recommend strategy echo "" echo "=== Recommended Strategy ===" if [[ "$has_reload" == "true" ]]; then echo " 1. Send reload signal: systemctl reload ${service}" echo " 2. Verify health: systemctl is-active --quiet ${service}" echo " 3. Check journal: journalctl -u ${service} --since '1 min ago' -n 20" else echo " 1. Backup current state: systemctl show ${service} > /tmp/${service}.state" echo " 2. Apply configuration changes" echo " 3. Restart service: systemctl restart ${service}" echo " 4. Wait for readiness: systemctl is-active --quiet ${service}" echo " 5. Verify health: journalctl -u ${service} --since '1 min ago'" echo " 6. If health fails: systemctl revert ${service} && systemctl restart ${service}" fi } determine_strategy "${1:?Usage: $0 <service.service>}" ``` **Checkpoint:** Strategy is determined and matches the service's actual capabilities. Reload is preferred over restart wherever possible. ### 2. Execute Zero-Downtime Configuration Change Apply changes using the service's native reload mechanism with active health verification. ```bash #!/usr/bin/env bash # zero_downtime_deploy.sh — Deploy configuration changes with zero-downtime guarantees # Usage: ./zero_downtime_deploy.sh <service.service> <config-changes-file> [--health-url http://localhost:8080/health] set -euo pipefail SERVICE_NAME="${1:?Usage: $0 <service.service> <config-changes-file> [--health-url URL]}" CHANGES_FILE="${2:?Config changes file required}" HEALTH_URL="${3:-}" DEPLOY_START=$(date +%s) ROLLBACK_NEEDED=false cleanup_on_failure() { if [[ "$ROLLBACK_NEEDED" == "true" ]]; then echo "=== ROLLBACK: Reverting configuration changes ===" if systemctl revert "$SERVICE_NAME" &>/dev/null; then systemctl restart "$SERVICE_NAME" echo "Configuration reverted and service restarted" else echo "ERROR: Rollback failed — manual intervention required" >&2 exit 1 fi fi } trap cleanup_on_failure ERR verify_service_health() { local service="$1" local max_attempts=10 local attempt=0 echo "--- Health Verification ---" # Method 1: systemd active state if ! systemctl is-active --quiet "$service"; then echo "FAIL: Service is not active" ROLLBACK_NEEDED=true return 1 fi echo " systemd active: OK" # Method 2: Health URL if provided if [[ -n "$HEALTH_URL" ]]; then while [[ $attempt -lt $max_attempts ]]; do if curl -sf --max-time 5 "$HEALTH_URL" &>/dev/null; then echo " health endpoint: OK" return 0 fi attempt=$((attempt + 1)) echo " health endpoint: waiting (attempt $attempt/$max_attempts)..." sleep 3 done echo " health endpoint: FAIL (timed out after $((max_attempts * 3))s)" ROLLBACK_NEEDED=true return 1 fi # Method 3: Journal check for errors after restart local recent_errors recent_errors=$(journalctl -u "$service" --since "1 minute ago" -p err --no-pager 2>/dev/null | wc -l) if [[ "$recent_errors" -gt 3 ]]; then echo " journal error count: WARNING (${recent_errors} errors in last minute)" echo " Consider monitoring for 60 more seconds" return 0 fi echo " journal error count: OK (${recent_errors} errors)" return 0 } echo "=== Zero-Downtime Deployment ===" echo "Service: $SERVICE_NAME" echo "Changes: $CHANGES_FILE" echo "Start: $(date -Iseconds)" echo "" # Verify service is running echo "--- Pre-change Verification ---" if ! systemctl is-active --quiet "$SERVICE_NAME"; then echo "ERROR: Service '$SERVICE_NAME' is not running" >&2 exit 1 fi echo "Service is active. Proceeding..." echo "" # Apply configuration changes safely echo "--- Applying Changes ---" if [[ -f "$CHANGES_FILE" ]]; then # If it's a conf.d drop-in, place it in the correct location local_unit_dir="/etc/systemd/system/${SERVICE_NAME}.d" mkdir -p "$local_unit_dir" # Generate a unique drop-in filename local timestamp timestamp=$(date +%Y%m%dT%H%M%S) local dropin_file="${local_unit_dir}/99-custom-${timestamp}.conf" if cp "$CHANGES_FILE" "$dropin_file"; then echo " Drop-in created: $dropin_file" else echo "ERROR: Failed to create drop-in file" >&2 exit 1 fi else echo "WARNING: Config file not found, skipping file deployment" fi echo "" # Reload systemd and service echo "--- Reloading Systemd ---" systemctl daemon-reload echo " systemd daemon reloaded" echo "--- Reloading Service ---" if systemctl reload "$SERVICE_NAME" &>/dev/null; then echo " Service reloaded successfully (graceful)" elif systemctl restart "$SERVICE_NAME" &>/dev/null; then echo " Service restarted (graceful reload unavailable)" ROLLBACK_NEEDED=true # Mark for rollback on health failure else echo "ERROR: Failed to reload/restart service" >&2 ROLLBACK_NEEDED=true exit 1 fi echo "" # Health verification verify_service_health "$SERVICE_NAME" ELAPSED=$(($(date +%s) - DEPLOY_START)) echo "" echo "=== Deployment Summary ===" echo "Service: $SERVICE_NAME" echo "Duration: ${ELAPSED}s" echo "Status: $(systemctl is-active "$SERVICE_NAME")" echo "Health: $(verify_service_health "$SERVICE_NAME" && echo 'PASS' || echo 'FAIL')" if [[ "$ROLLBACK_NEEDED" == "true" ]]; then echo "WARNING: Changes marked for rollback on health failure" else echo "Changes committed. No rollback needed." fi ``` **Checkpoint:** Service is active and healthy after changes. All configuration modifications are verified with active health checks. ### 3. Implement Health Monitoring During Changes Set up continuous health monitoring during the change window with automated alerting on degradation. ```python """Health monitor for zero-downtime service changes. Provides active health checking with configurable thresholds, alerting on degradation during change windows. """ import subprocess import time import json import urllib.request from dataclasses import dataclass, field from datetime import datetime, timedelta from enum import Enum from typing import Optional class HealthStatus(Enum): HEALTHY = "healthy" DEGRADED = "degraded" UNHEALTHY = "unhealthy" UNKNOWN = "unknown" @dataclass class HealthMetric: """A single health metric sample.""" timestamp: datetime status: HealthStatus response_time_ms: Optional[float] = None error_count: int = 0 detail: str = "" @dataclass class HealthMonitorConfig: """Configuration for a health monitoring session.""" service_name: str check_interval: float = 2.0 # seconds between health checks min_healthy_samples: int = 5 # consecutive healthy checks to consider stable max_error_count: int = 3 # errors before marking unhealthy timeout: float = 5.0 # HTTP request timeout in seconds health_url: Optional[str] = None # HTTP health endpoint alert_on_degradation: bool = True # alert when status drops from healthy @dataclass class HealthMonitor: """Monitors a systemd service's health during a change window.""" config: HealthMonitorConfig history: list = field(default_factory=list) current_status: HealthStatus = HealthStatus.UNKNOWN degradation_detected: bool = False alerts: list = field(default_factory=list) def _systemd_health_check(self) -> tuple[HealthStatus, int]: """Check health via systemd active state. Returns: Tuple of (status, uptime_seconds) """ try: result = subprocess.run( ["systemctl", "is-active", "--quiet", self.config.service_name], capture_output=True, text=True, timeout=5 ) if result.returncode == 0: # Service is active — check uptime for stability uptime_result = subprocess.run( ["systemctl", "show", self.config.service_name, "--property=ActiveEnterTimestamp"], capture_output=True, text=True, timeout=5 ) uptime_line = uptime_result.stdout.strip().split("=", 1) if len(uptime_line) == 2: return HealthStatus.HEALTHY, 0 return HealthStatus.HEALTHY, 0 else: return HealthStatus.UNHEALTHY, 0 except subprocess.TimeoutExpired: return HealthStatus.UNKNOWN, 0 except FileNotFoundError: return HealthStatus.UNKNOWN, 0 def _http_health_check(self) -> tuple[HealthStatus, float]: """Check health via HTTP endpoint. Returns: Tuple of (status, response_time_ms) """ if not self.config.health_url: return HealthStatus.UNKNOWN, 0.0 start = time.monotonic() try: req = urllib.request.Request(self.config.health_url, method='GET') with urllib.request.urlopen(req, timeout=self.config.timeout) as resp: response_time = (time.monotonic() - start) * 1000