Executes Linux system changes and deployments using zero-downtime patterns, health monitoring, and safe rollback procedures to maintain host availability.
Executes Linux system changes and deployments using zero-downtime patterns, health monitoring, and safe rollback procedures to maintain host availability.
{"version":"1.0.0","domain":"linux","triggers":"zero downtime, service integrity, safe restart, rolling updates, health checks, Linux operational availability, service disruption prevention","role":"implementation","scope":"implementation","output-format":"code","content-types":["code","guidance","do-dont","examples"]}
Linux Service Integrity Operations
Infrastructure engineer executing zero-downtime changes and deployments on Linux hosts using safe restart patterns, health monitoring during transitions, and automatic rollback procedures to maintain continuous service availability.
TL;DR Checklist
Confirm the service supports graceful reload before considering restart
Set up a health check endpoint (HTTP /proc, or systemd-ready notification) before making changes
Verify rollback procedure and snapshot exist before starting the change
Execute the change with service-isolation (socket activation or port-based switching)
Monitor health for at least 60 seconds post-change with active health checks
Confirm dependent services have not entered degraded state
Document the change in runbook with before/after metrics
When to Use
Use this skill when:
Deploying configuration changes to production services — Applying new settings without unplanned downtime
Performing kernel or library updates on running systems — Services must remain available during package updates
Upgrading application binaries or containers — Zero-downtime deployment of new versions
Modifying firewall rules or network configurations — Maintaining active connections during policy changes
Scaling services vertically — Adjusting resource limits, CPU/memory quotas without service interruption
Applying security patches to production workloads — CVE fixes must not cause service outages
When NOT to Use
Avoid this skill for:
Service-level failures where the service is already down — Use incident response procedures instead
Network-level re-architecting (VPC changes, subnet migrations) — These require network planning and maintenance windows
Complete host re-imaging or OS upgrades — Requires planned downtime and rebuild procedures
Use linux-safe-script-execution for pre-flight analysis before initiating zero-downtime operations.
Core Workflow
1. Determine Service Restart Strategy
Choose the correct restart strategy based on the service's capabilities and dependency requirements.
#!/usr/bin/env bash# service_strategy.sh — Determine optimal change strategy for a systemd service# Usage: ./service_strategy.sh <service.service>set -euo pipefail
determine_strategy() {
local service="$1"if ! systemctl is-active --quiet "$service" 2>/dev/null; thenecho"ERROR: Service '$service' is not running" >&2
return 1
fiecho"=== Change Strategy Analysis: ${service} ==="echo""# Check if service supports reloadlocal has_reload=falselocal exec_start
exec_start=$(systemctl show "$service" --property=ExecStart --value 2>/dev/null || echo"")
# Check unit file for ExecReload directivelocal unit_file
unit_file=$(systemctl show "$service" --property=ExecReload --value 2>/dev/null || echo"")
if [[ -n "$unit_file" && "$unit_file" != "-" ]]; then
has_reload=trueecho"Strategy: GRACEFUL RELOAD (preferred)"echo" ExecReload directive found: $unit_file"echo" Reload sends SIGHUP — connections maintained, no downtime"elseecho" No ExecReload directive found — restart required"fiecho""# Check service type for readiness signalinglocal svc_type
svc_type=$(systemctl show "$service" --property=Type --value 2>/dev/null || echo"simple")
echo"Service type: $svc_type"case"$svc_type"in
notify)
echo" Uses sd_notify() — systemd tracks readiness accurately"echo" Strategy: Signal READY after config load completes"
;;
forking)
echo" Forking process — systemd waits for parent exit"echo" Strategy: Fork handles config reload, parent exits"
;;
oneshot)
echo" One-shot service — not a long-running daemon"echo" Strategy: Re-run the oneshot with new parameters"
;;
simple|exec|dbus|idle)
echo" Simple/exec service — process is the main process"echo" Strategy: Send signal to main process directly"
;;
esacecho""# Check for socket activationlocal socket_unit
socket_unit=$(systemctl list-dependencies "$service" --reverse --no-pager 2>/dev/null | grep '\.socket' | head -5 || true)
if [[ -n "$socket_unit" ]]; thenecho"Socket activation detected:"echo"$socket_unit" | sed 's/^/ /'echo" Benefit: systemd holds listening sockets during restart"echo" New connections queue until service is ready"elseecho" No socket activation — brief connection gap during restart"fiecho""# Determine health check methodecho"=== Health Check Methods ==="if grep -q "ExecStartPre=""$unit_file" 2>/dev/null || grep -q "ExecStart=""$unit_file" 2>/dev/null; thenlocal health_url
health_url=$(grep -iE "(health|ready|status|ping)""$unit_file" 2>/dev/null | grep -iE "exec|cmd" || true)
if [[ -n "$health_url" ]]; thenecho" Built-in health endpoint found"elseecho" No explicit health endpoint — use systemctl is-active"fifi# Recommend strategyecho""echo"=== Recommended Strategy ==="if [[ "$has_reload" == "true" ]]; thenecho" 1. Send reload signal: systemctl reload ${service}"echo" 2. Verify health: systemctl is-active --quiet ${service}"echo" 3. Check journal: journalctl -u ${service} --since '1 min ago' -n 20"elseecho" 1. Backup current state: systemctl show ${service} > /tmp/${service}.state"echo" 2. Apply configuration changes"echo" 3. Restart service: systemctl restart ${service}"echo" 4. Wait for readiness: systemctl is-active --quiet ${service}"echo" 5. Verify health: journalctl -u ${service} --since '1 min ago'"echo" 6. If health fails: systemctl revert ${service} && systemctl restart ${service}"fi
}
determine_strategy "${1:?Usage: $0 <service.service>}"
Checkpoint: Strategy is determined and matches the service's actual capabilities. Reload is preferred over restart wherever possible.
2. Execute Zero-Downtime Configuration Change
Apply changes using the service's native reload mechanism with active health verification.
#!/usr/bin/env bash# zero_downtime_deploy.sh — Deploy configuration changes with zero-downtime guarantees# Usage: ./zero_downtime_deploy.sh <service.service> <config-changes-file> [--health-url http://localhost:8080/health]set -euo pipefail
SERVICE_NAME="${1:?Usage: $0 <service.service> <config-changes-file> [--health-url URL]}"
CHANGES_FILE="${2:?Config changes file required}"
HEALTH_URL="${3:-}"
DEPLOY_START=$(date +%s)
ROLLBACK_NEEDED=falsecleanup_on_failure() {
if [[ "$ROLLBACK_NEEDED" == "true" ]]; thenecho"=== ROLLBACK: Reverting configuration changes ==="if systemctl revert "$SERVICE_NAME" &>/dev/null; then
systemctl restart "$SERVICE_NAME"echo"Configuration reverted and service restarted"elseecho"ERROR: Rollback failed — manual intervention required" >&2
exit 1
fifi
}
trap cleanup_on_failure ERR
verify_service_health() {
local service="$1"local max_attempts=10
local attempt=0
echo"--- Health Verification ---"# Method 1: systemd active stateif ! systemctl is-active --quiet "$service"; thenecho"FAIL: Service is not active"
ROLLBACK_NEEDED=truereturn 1
fiecho" systemd active: OK"# Method 2: Health URL if providedif [[ -n "$HEALTH_URL" ]]; thenwhile [[ $attempt -lt $max_attempts ]]; doif curl -sf --max-time 5 "$HEALTH_URL" &>/dev/null; thenecho" health endpoint: OK"return 0
fi
attempt=$((attempt + 1))
echo" health endpoint: waiting (attempt $attempt/$max_attempts)..."sleep 3
doneecho" health endpoint: FAIL (timed out after $((max_attempts * 3))s)"
ROLLBACK_NEEDED=truereturn 1
fi# Method 3: Journal check for errors after restartlocal recent_errors
recent_errors=$(journalctl -u "$service" --since "1 minute ago" -p err --no-pager 2>/dev/null | wc -l)
if [[ "$recent_errors" -gt 3 ]]; thenecho" journal error count: WARNING (${recent_errors} errors in last minute)"echo" Consider monitoring for 60 more seconds"return 0
fiecho" journal error count: OK (${recent_errors} errors)"return 0
}
echo"=== Zero-Downtime Deployment ==="echo"Service: $SERVICE_NAME"echo"Changes: $CHANGES_FILE"echo"Start: $(date -Iseconds)"echo""# Verify service is runningecho"--- Pre-change Verification ---"if ! systemctl is-active --quiet "$SERVICE_NAME"; thenecho"ERROR: Service '$SERVICE_NAME' is not running" >&2
exit 1
fiecho"Service is active. Proceeding..."echo""# Apply configuration changes safelyecho"--- Applying Changes ---"if [[ -f "$CHANGES_FILE" ]]; then# If it's a conf.d drop-in, place it in the correct location
local_unit_dir="/etc/systemd/system/${SERVICE_NAME}.d"mkdir -p "$local_unit_dir"# Generate a unique drop-in filenamelocal timestamp
timestamp=$(date +%Y%m%dT%H%M%S)
local dropin_file="${local_unit_dir}/99-custom-${timestamp}.conf"ifcp"$CHANGES_FILE""$dropin_file"; thenecho" Drop-in created: $dropin_file"elseecho"ERROR: Failed to create drop-in file" >&2
exit 1
fielseecho"WARNING: Config file not found, skipping file deployment"fiecho""# Reload systemd and serviceecho"--- Reloading Systemd ---"
systemctl daemon-reload
echo" systemd daemon reloaded"echo"--- Reloading Service ---"if systemctl reload "$SERVICE_NAME" &>/dev/null; thenecho" Service reloaded successfully (graceful)"elif systemctl restart "$SERVICE_NAME" &>/dev/null; thenecho" Service restarted (graceful reload unavailable)"
ROLLBACK_NEEDED=true# Mark for rollback on health failureelseecho"ERROR: Failed to reload/restart service" >&2
ROLLBACK_NEEDED=trueexit 1
fiecho""# Health verification
verify_service_health "$SERVICE_NAME"
ELAPSED=$(($(date +%s) - DEPLOY_START))
echo""echo"=== Deployment Summary ==="echo"Service: $SERVICE_NAME"echo"Duration: ${ELAPSED}s"echo"Status: $(systemctl is-active "$SERVICE_NAME")"echo"Health: $(verify_service_health "$SERVICE_NAME" && echo 'PASS' || echo 'FAIL')"if [[ "$ROLLBACK_NEEDED" == "true" ]]; thenecho"WARNING: Changes marked for rollback on health failure"elseecho"Changes committed. No rollback needed."fi
Checkpoint: Service is active and healthy after changes. All configuration modifications are verified with active health checks.
3. Implement Health Monitoring During Changes
Set up continuous health monitoring during the change window with automated alerting on degradation.
"""Health monitor for zero-downtime service changes.
Provides active health checking with configurable thresholds,
alerting on degradation during change windows.
"""import subprocess
import time
import json
import urllib.request
from dataclasses import dataclass, field
from datetime import datetime, timedelta
from enum import Enum
from typing importOptionalclassHealthStatus(Enum):
HEALTHY = "healthy"
DEGRADED = "degraded"
UNHEALTHY = "unhealthy"
UNKNOWN = "unknown"@dataclassclassHealthMetric:
"""A single health metric sample."""
timestamp: datetime
status: HealthStatus
response_time_ms: Optional[float] = None
error_count: int = 0
detail: str = ""@dataclassclassHealthMonitorConfig:
"""Configuration for a health monitoring session."""
service_name: str
check_interval: float = 2.0# seconds between health checks
min_healthy_samples: int = 5# consecutive healthy checks to consider stable
max_error_count: int = 3# errors before marking unhealthy
timeout: float = 5.0# HTTP request timeout in seconds
health_url: Optional[str] = None# HTTP health endpoint
alert_on_degradation: bool = True# alert when status drops from healthy@dataclassclassHealthMonitor:
"""Monitors a systemd service's health during a change window."""
config: HealthMonitorConfig
history: list = field(default_factory=list)
current_status: HealthStatus = HealthStatus.UNKNOWN
degradation_detected: bool = False
alerts: list = field(default_factory=list)
def_systemd_health_check(self) -> tuple[HealthStatus, int]:
"""Check health via systemd active state.
Returns:
Tuple of (status, uptime_seconds)
"""try:
result = subprocess.run(
["systemctl", "is-active", "--quiet", self.config.service_name],
capture_output=True, text=True, timeout=5
)
if result.returncode == 0:
# Service is active — check uptime for stability
uptime_result = subprocess.run(
["systemctl", "show", self.config.service_name,
"--property=ActiveEnterTimestamp"],
capture_output=True, text=True, timeout=5
)
uptime_line = uptime_result.stdout.strip().split("=", 1)
iflen(uptime_line) == 2:
return HealthStatus.HEALTHY, 0return HealthStatus.HEALTHY, 0else:
return HealthStatus.UNHEALTHY, 0except subprocess.TimeoutExpired:
return HealthStatus.UNKNOWN, 0except FileNotFoundError:
return HealthStatus.UNKNOWN, 0def_http_health_check(self) -> tuple[HealthStatus, float]:
"""Check health via HTTP endpoint.
Returns:
Tuple of (status, response_time_ms)
"""ifnotself.config.health_url:
return HealthStatus.UNKNOWN, 0.0
start = time.monotonic()
try:
req = urllib.request.Request(self.config.health_url, method='GET')
with urllib.request.urlopen(req, timeout=self.config.timeout) as resp:
response_time = (time.monotonic() - start) * 1000if200 <= resp.status < 300:
return HealthStatus.HEALTHY, response_time
elif resp.status < 500:
return HealthStatus.DEGRADED, response_time
else:
return HealthStatus.UNHEALTHY, response_time
except (urllib.error.URLError, urllib.error.HTTPError) as e:
response_time = (time.monotonic() - start) * 1000ifisinstance(e, urllib.error.HTTPError):
if500 <= e.code < 600:
return HealthStatus.UNHEALTHY, response_time
return HealthStatus.DEGRADED, response_time
return HealthStatus.UNHEALTHY, response_time
except Exception:
return HealthStatus.UNKNOWN, 0.0defperform_check(self) -> HealthMetric:
"""Perform a single health check and record the result.
Returns:
HealthMetric with the check result
"""
start = time.monotonic()
# Use HTTP check if available, otherwise systemdifself.config.health_url:
status, response_time = self._http_health_check()
else:
status, _ = self._systemd_health_check()
response_time = 0.0
elapsed_ms = (time.monotonic() - start) * 1000# Update degradation tracking
prev_status = self.current_status
self.current_status = status
if prev_status == HealthStatus.HEALTHY and status in (
HealthStatus.DEGRADED, HealthStatus.UNHEALTHY
):
self.degradation_detected = True
alert_msg = (
f"Degradation detected for {self.config.service_name}: "f"{prev_status.value} → {status.value} at {datetime.now().isoformat()}"
)
self.alerts.append(alert_msg)
metric = HealthMetric(
timestamp=datetime.now(),
status=status,
response_time_ms=response_time or elapsed_ms
)
self.history.append(metric)
return metric
defrun_monitoring_session(
self,
duration_seconds: int = 120,
callback=None) -> list[HealthMetric]:
"""Run health monitoring for a specified duration.
Args:
duration_seconds: How long to monitor
callback: Optional function(metric) called after each check
Returns:
List of all HealthMetric samples collected
"""
end_time = time.monotonic() + duration_seconds
healthy_streak = 0while time.monotonic() < end_time:
metric = self.perform_check()
if callback:
callback(metric)
if metric.status == HealthStatus.HEALTHY:
healthy_streak += 1else:
healthy_streak = 0# Wait for next check interval
time.sleep(self.config.check_interval)
# Final status
stability = (
"STABLE"if healthy_streak >= self.config.min_healthy_samples
else"UNSTABLE"
)
print(f"\n{'='*50}")
print(f"Monitoring Session Complete")
print(f"Service: {self.config.service_name}")
print(f"Total checks: {len(self.history)}")
print(f"Stability: {stability}")
print(f"Final status: {self.current_status.value}")
ifself.alerts:
print(f"Alerts: {len(self.alerts)}")
for alert inself.alerts:
print(f" ⚠ {alert}")
print(f"{'='*50}")
returnself.history
defgenerate_report(self) -> dict:
"""Generate a health monitoring summary report.
Returns:
Dictionary with monitoring summary data
"""ifnotself.history:
return {"error": "No monitoring data collected"}
statuses = [m.status for m inself.history]
healthy_count = statuses.count(HealthStatus.HEALTHY)
degraded_count = statuses.count(HealthStatus.DEGRADED)
unhealthy_count = statuses.count(HealthStatus.UNHEALTHY)
total = len(statuses)
avg_response = (
sum(m.response_time_ms or0for m inself.history) / total
if total > 0else0
)
return {
"service": self.config.service_name,
"total_checks": total,
"healthy_pct": round(healthy_count / total * 100, 1) if total else0,
"degraded_pct": round(degraded_count / total * 100, 1) if total else0,
"unhealthy_pct": round(unhealthy_count / total * 100, 1) if total else0,
"avg_response_ms": round(avg_response, 1),
"degradation_detected": self.degradation_detected,
"alerts": self.alerts,
"final_status": self.current_status.value,
"samples": [
{
"time": m.timestamp.isoformat(),
"status": m.status.value,
"response_ms": round(m.response_time_ms or0, 1),
}
for m inself.history[-10:] # Last 10 samples
],
}
Checkpoint: Health monitoring runs for a minimum of 120 seconds post-change. Service achieves min_healthy_samples consecutive healthy checks before declaring stability.
4. Execute Safe Rollback
If health checks fail during or after a change, execute an automated rollback to restore the previous state.
#!/usr/bin/env bash# safe_rollback.sh — Automated rollback for zero-downtime deployments# Usage: ./safe_rollback.sh <service.service> [--verbose]set -euo pipefail
SERVICE_NAME="${1:?Usage: $0 <service.service> [--verbose]}"
VERBOSE=false
ROLLBACK_TIMESTAMP=$(date +%Y%m%dT%H%M%S)
if [[ "${2:-}" == "--verbose" ]]; then
VERBOSE=truefilog() {
if [[ "$VERBOSE" == "true" ]]; thenecho"[$(date +%T)] $*"fi
}
echo"=== Service Rollback: ${SERVICE_NAME} ==="echo"Timestamp: $(date -Iseconds)"echo""# Phase 1: Capture current (broken) stateecho"--- Phase 1: Capture Current State ---"
systemctl show "$SERVICE_NAME" --property=LoadState,ActiveState,SubState \
> "/tmp/rollback-${SERVICE_NAME}-${ROLLBACK_TIMESTAMP}-pre.conf" 2>/dev/null || truelog"Current service state saved"# Phase 2: Revert systemd drop-ins and configurationecho"--- Phase 2: Revert Configuration ---"# Find and remove all custom drop-ins created after the deployment
local_unit_dir="/etc/systemd/system/${SERVICE_NAME}.d"if [[ -d "$local_unit_dir" ]]; then# List all custom drop-ins (those not in version control or baseline)local dropin_count
dropin_count=$(find "$local_unit_dir" -name "*.conf" 2>/dev/null | wc -l)
if [[ "$dropin_count" -gt 0 ]]; then# Keep only baseline drop-ins (files without timestamp in name)
find "$local_unit_dir" -name "*.conf" -type f | while IFS= read -r conf; dolocalbasenamebasename=$(basename"$conf")
if [[ "$basename" == *-T* ]]; then# Timestamp-based drop-in — remove itrm -f "$conf"log"Removed drop-in: $basename"fidonefi# Use systemd's native revert if available (systemd 252+)if systemctl revert "$SERVICE_NAME" 2>/dev/null; thenlog"systemd native revert successful"elselog"systemd revert unavailable — using manual config restore"fielselog"No custom drop-in directory found"fiecho""# Phase 3: Reload and restartecho"--- Phase 3: Reload and Restart ---"
systemctl daemon-reload
log"systemd daemon reloaded"# Attempt graceful restartif systemctl restart "$SERVICE_NAME" 2>/dev/null; thenlog"Service restarted successfully"elselog"WARNING: Service restart failed"fiecho""# Phase 4: Verify rollback healthecho"--- Phase 4: Verify Rollback ---"sleep 3
if systemctl is-active --quiet "$SERVICE_NAME"; thenecho" Service is ACTIVE after rollback"# Check journal for errors in the last minutelocal error_count
error_count=$(journalctl -u "$SERVICE_NAME" --since "1 minute ago" \
-p err --no-pager 2>/dev/null | wc -l)
if [[ "$error_count" -gt 0 ]]; thenecho" WARNING: ${error_count} error(s) in journal since restart"
journalctl -u "$SERVICE_NAME" --since "1 minute ago" -p err --no-pager -n 10
elseecho" No errors in journal since restart — rollback verified"fielseecho" ERROR: Service is NOT active after rollback"echo" Attempting one more restart..."
systemctl restart "$SERVICE_NAME" 2>/dev/null || truesleep 3
if ! systemctl is-active --quiet "$SERVICE_NAME"; thenecho" CRITICAL: Service remains down after rollback"echo" Manual intervention required"exit 1
fifiecho""# Phase 5: Preserve rollback artifactsecho"--- Phase 5: Preserve Rollback Artifacts ---"local rollback_dir="/var/log/rollback/${SERVICE_NAME}"mkdir -p "$rollback_dir"# Copy current service state for post-mortem
systemctl show "$SERVICE_NAME" > "${rollback_dir}/post-rollback.conf"
journalctl -u "$SERVICE_NAME" --since "${ROLLBACK_TIMESTAMP}" \
> "${rollback_dir}/journal.log" 2>/dev/null || trueecho" Artifacts preserved in: $rollback_dir"echo""echo"=== Rollback Complete ==="echo"Service: $SERVICE_NAME"echo"Status: $(systemctl is-active "$SERVICE_NAME")"
Checkpoint: Service is confirmed active after rollback. Artifacts are preserved for post-mortem analysis.
5. Safe Rolling Update for Multi-Instance Services
For services running multiple instances (via systemd templates or containers), perform rolling updates to maintain availability.
#!/usr/bin/env bash# rolling_update.sh — Perform rolling updates for multi-instance services# Usage: ./rolling_update.sh <template.service> <instances_count> [--health-url URL]set -euo pipefail
TEMPLATE_SERVICE="${1:?Usage: $0 <template.service> <count> [--health-url URL]}"
INSTANCE_COUNT="${2:?Instance count required}"
HEALTH_URL="${3:-}"
MAX_UNHEALTHY=1 # Maximum instances allowed to be down simultaneously
UPDATE_LOG="/var/log/rolling-update-${TEMPLATE_SERVICE}-$(date +%Y%m%d).log"mkdir -p "$(dirname "$UPDATE_LOG")"log() {
echo"[$(date +%T)] $*" | tee -a "$UPDATE_LOG"
}
check_instance_health() {
local instance="$1"if systemctl is-active --quiet "$instance" 2>/dev/null; thenreturn 0
fireturn 1
}
wait_for_stability() {
local instance="$1"local max_wait=30
localwait=0
log"Waiting for $instance to stabilize..."while [[ $wait -lt $max_wait ]]; doif check_instance_health "$instance"; thenlog"$instance is healthy after ${wait}s"return 0
fisleep 2
wait=$((wait + 2))
donelog"ERROR: $instance did not stabilize within ${max_wait}s"return 1
}
log"=== Rolling Update: ${TEMPLATE_SERVICE} ==="log"Instances: $INSTANCE_COUNT"log"Max down simultaneously: $MAX_UNHEALTHY"log""# Phase 1: Pre-update health checklog"--- Pre-update Health Check ---"for i in $(seq 1 "$INSTANCE_COUNT"); dolocal instance="${TEMPLATE_SERVICE/@/$i}"if check_instance_health "$instance"; thenlog" Instance $i: HEALTHY"elselog" Instance $i: UNHEALTHY (may need separate attention)"fidonelog""# Phase 2: Rolling updatelog"--- Rolling Update ---"for i in $(seq 1 "$INSTANCE_COUNT"); dolocal instance="${TEMPLATE_SERVICE/@/$i}"local down_count=0
log"Updating instance $i of $INSTANCE_COUNT: $instance"# Stop the current instance
systemctl stop "$instance" 2>/dev/null
log" Instance $i stopped"# Wait briefly for port cleanupsleep 1
# Verify other instances are still healthyfor j in $(seq 1 "$INSTANCE_COUNT"); doif [[ $j -ne $i ]]; thenlocal other="${TEMPLATE_SERVICE/@/$j}"if ! check_instance_health "$other"; then
down_count=$((down_count + 1))
fifidoneif [[ $down_count -gt $MAX_UNHEALTHY ]]; thenlog" ERROR: Too many instances unhealthy ($down_count > $MAX_UNHEALTHY) — aborting"# Attempt to restart the stopped instance immediately
systemctl start "$instance" 2>/dev/null
log" Aborting rolling update — restart instance $i for recovery"exit 1
fi# Start the instance
systemctl start "$instance" 2>/dev/null
log" Instance $i started"# Wait for stabilityif ! wait_for_stability "$instance"; thenlog" ERROR: Instance $i failed to stabilize"log" Attempting restart..."
systemctl restart "$instance" 2>/dev/null
wait_for_stability "$instance" || truefilog" Instance $i: DEPLOYED"log""done# Phase 3: Post-update verificationlog"--- Post-update Verification ---"
all_healthy=truefor i in $(seq 1 "$INSTANCE_COUNT"); dolocal instance="${TEMPLATE_SERVICE/@/$i}"if check_instance_health "$instance"; thenlog" Instance $i: HEALTHY"elselog" Instance $i: UNHEALTHY — manual attention required"
all_healthy=falsefidonelog""if [[ "$all_healthy" == "true" ]]; thenlog"=== Rolling Update Complete: All instances healthy ==="elselog"=== Rolling Update Complete: Some instances need attention ==="exit 1
fi
Checkpoint: All instances are healthy after rolling update. No more than MAX_UNHEALTHY instances were down simultaneously.
Implementation Patterns
Pattern 1: Graceful Shutdown vs Hard Restart (BAD vs. GOOD)
BAD — Hard restart causes service disruption
# ❌ BAD: Direct restart with no signal handling or health verificationdeploy_update() {
local service="$1"# Stops immediately — active connections are dropped
systemctl restart "$service"# No health check — assumes successecho"Update complete"
}
# Problems:# - All active connections are dropped during restart# - No health verification — service might fail to start# - No rollback if health check fails# - Dependent services start before this one is ready
GOOD — Graceful transition with health verification
# ✅ GOOD: Graceful shutdown, health verification, and automatic rollbackdeploy_update() {
local service="$1"local max_health_checks=15
local health_check_interval=3
local rollback=false# Step 1: Check if service supports reload (zero-downtime preferred)local has_reload
has_reload=$(systemctl show "$service" --property=ExecReload --value 2>/dev/null)
if [[ -n "$has_reload" && "$has_reload" != "-" ]]; thenecho"Using graceful reload (zero downtime)"
systemctl reload "$service"
rollback=false# Reload rarely needs rollbackelse# Step 2: Graceful stop — send SIGTERM, not SIGKILLecho"Graceful stop (connections draining)..."
systemctl stop "$service"
rollback=true# Restart needs rollback on failure# Step 3: Quick health verificationecho"Verifying service health..."local checks=0
while [[ $checks -lt $max_health_checks ]]; doif systemctl is-active --quiet "$service" 2>/dev/null; thenecho"Service started and healthy after ${checks}s"return 0
fi
checks=$((checks + 1))
echo" Waiting for service... (${checks}/${max_health_checks})"sleep"$health_check_interval"done# Step 4: Rollback on failureecho"ERROR: Service failed to start" >&2
if [[ "$rollback" == "true" ]]; thenecho"Rolling back..."
systemctl start "$service" 2>/dev/null || trueecho"Rollback complete. Manual review required."fireturn 1
fi# Step 5: Post-deploy verificationif systemctl is-active --quiet "$service"; thenecho"Service healthy after deployment"return 0
elseecho"ERROR: Service unhealthy after deployment" >&2
return 1
fi
}
Pattern 2: Health Check with Backoff and Circuit Breaker
Bash — Resilient health monitoring with circuit breaker pattern
#!/usr/bin/env bash# resilient_health_monitor.sh — Health check with exponential backoff and circuit breaker# Usage: ./resilient_health_monitor.sh <service.service> [--http URL] [--max-checks 30]set -euo pipefail
SERVICE_NAME="${1:?Usage: $0 <service.service> [--http URL] [--max-checks N]}"
HEALTH_URL=""
MAX_CHECKS=30
CONSECUTIVE_HEALTHY=0
CIRCUIT_BREAKER_THRESHOLD=3 # consecutive failures to open circuit
CIRCUIT_OPEN=false
RETRY_BACKOFF_BASE=2
while [[ $# -gt 0 ]]; docase"$1"in
--http) HEALTH_URL="$2"; shift 2 ;;
--max-checks) MAX_CHECKS="$2"; shift 2 ;;
*) shift ;;
esacdoneperform_health_check() {
if [[ -n "$HEALTH_URL" ]]; then# HTTP health check with timeoutif curl -sf --max-time 3 "$HEALTH_URL" &>/dev/null; thenreturn 0
elsereturn 1
fielse# Systemd health checkif systemctl is-active --quiet "$SERVICE_NAME" 2>/dev/null; thenreturn 0
elsereturn 1
fifi
}
echo"=== Resilient Health Monitor ==="echo"Service: $SERVICE_NAME"echo"Max checks: $MAX_CHECKS"echo"Circuit breaker threshold: $CIRCUIT_BREAKER_THRESHOLD"echo""
check_num=0
while [[ $check_num -lt $MAX_CHECKS ]]; do
check_num=$((check_num + 1))
if [[ "$CIRCUIT_OPEN" == "true" ]]; then# Circuit is open — wait for half-open probeecho"[$check_num] Circuit OPEN — waiting for recovery probe..."sleep 10
# Half-open: attempt one health checkif perform_health_check; thenecho" Circuit HALF-OPEN: service recovered"
CIRCUIT_OPEN=false
CONSECUTIVE_HEALTHY=1
elseecho" Circuit stays OPEN: service still unhealthy"ficontinuefi# Normal health checkif perform_health_check; then
CONSECUTIVE_HEALTHY=$((CONSECUTIVE_HEALTHY + 1))
backoff=$((RETRY_BACKOFF_BASE ** (CONSECUTIVE_HEALTHY / 5))) # Increase interval every 5 healthy checksif [[ $CONSECUTIVE_HEALTHY -ge 5 ]]; thenecho"[$check_num] ✓ HEALTHY (consecutive: $CONSECUTIVE_HEALTHY, next check in ${backoff}s)"elseecho"[$check_num] ✓ HEALTHY (consecutive: $CONSECUTIVE_HEALTHY)"fi# Sleep with backoff between checkssleep"$backoff"else
CONSECUTIVE_HEALTHY=0
if [[ $check_num -le 3 ]]; then# Initial transient failures — retry quicklyecho"[$check_num] ✗ UNHEALTHY (transient, retrying...)"sleep 2
elseecho"[$check_num] ✗ UNHEALTHY"# Check journal for errorslocal errors
errors=$(journalctl -u "$SERVICE_NAME" --since "${check_num} minutes ago" \
-p err --no-pager 2>/dev/null | tail -3 || true)
if [[ -n "$errors" ]]; thenecho" Recent errors:"echo"$errors" | sed 's/^/ /'fi# Circuit breaker: if too many consecutive failures, open circuitif [[ $check_num -ge $CIRCUIT_BREAKER_THRESHOLD ]]; then# Only open circuit after the first few checks are healthyif [[ $CONSECUTIVE_HEALTHY -le 1 ]]; thenecho" ⚠ Circuit OPEN: too many failures"
CIRCUIT_OPEN=truefifisleep 5
fifidoneecho""echo"=== Monitor Session Complete ==="if [[ "$CIRCUIT_OPEN" == "true" ]]; thenecho"Circuit remains OPEN — manual intervention required"exit 1
elseecho"Circuit closed — service is healthy"exit 0
fi
Constraints
MUST DO
MUST attempt systemctl reload before systemctl restart — reload maintains active connections and provides true zero-downtime
MUST run health checks for a minimum of 120 seconds after any service change — short-lived stability does not indicate lasting availability
MUST maintain a rollback-ready state before every change: configuration backup, snapshot, or systemd revert capability
MUST use a circuit breaker pattern in health monitoring — open circuit after consecutive failures to prevent check storms
MUST enforce MAX_UNHEALTHY=1 for rolling updates — never take down more than one instance simultaneously
MUST verify that dependent services have not entered a degraded state after the target service changes
MUST use socket activation where possible — systemd's socket listeners hold connections during service restarts
MUST preserve rollback artifacts (journal logs, service state, config diffs) for post-deploy analysis
MUST NOT DO
MUST NOT send SIGKILL or use systemctl kill --signal=9 to restart a service — use graceful shutdown with SIGTERM first
MUST NOT skip the health check phase regardless of how confident the change is — every change has failure risk
MUST NOT perform rolling updates on single-instance services — use the full deployment workflow with rollback instead
MUST NOT disable or bypass the circuit breaker "temporarily" — it prevents cascading failure detection
MUST NOT restart core infrastructure services (systemd-journald, dbus, NetworkManager) on production hosts without console access
MUST NOT modify firewall rules without an active management session (SSH) to restore them if the change locks you out
MUST NOT assume a service started correctly because systemctl restart returned 0 — always verify actual health state
Output Template
When applying this skill, produce:
Strategy Recommendation — Reload vs restart vs rolling update, with justification based on service type and dependencies
Pre-Change Health Baseline — Current service health metrics (active status, error count, response times if available)
Deployment Execution Plan — Step-by-step commands with timing estimates and expected behavior at each step
Health Verification Report — Post-change health check results with pass/fail status and duration
Rollback Readiness Statement — Confirmation that rollback is prepared with specific artifacts and recovery steps
Post-Change Stability Assessment — 120+ second health monitoring summary with degradation alerts and circuit breaker status
Related Skills
Skill
Purpose
linux-safe-script-execution
Pre-flight validation before initiating zero-downtime operations
linux-services
Deep systemd unit file configuration for socket activation and watchdog
observability
Set up Prometheus/Grafana health monitoring and alerting for service changes
networking
Network port management and load balancer integration for rolling updates
Live References
Authoritative documentation links for this skill's domain. The model follows markdown links at load time to resolve external references and inline content.
