zero-downtime-deploy

name	zero-downtime-deploy
disable-model-invocation	true
description	Zero-downtime deployment strategies and automation: blue-green, rolling, and canary deployments with Nginx upstream switching, symlink-based atomic deploys, rsync deploy scripts, GitHub Actions SSH deploy workflows, health check polling, and rollback runbooks. USE WHEN: - Implementing blue-green deployment with Nginx upstream swapping - Writing a production deploy script with symlink-based atomic release management - Configuring canary traffic splitting with Nginx split_clients - Setting up a GitHub Actions workflow that deploys via SSH with health check polling - Writing or testing a rollback runbook - Planning database migration strategy alongside zero-downtime deploys DO NOT USE FOR: - Kubernetes rolling updates and canary with Argo Rollouts (use the kubernetes skill) - Container image building and registry management (use the docker skill) - Cloud-specific deployment (AWS CodeDeploy, GCP Cloud Deploy — use the cloud skill) - Frontend-only CDN cache invalidation strategies
allowed-tools	Read, Grep, Glob, Write, Edit, Bash

Zero-Downtime Deployment — Strategies and Automation

Strategy Comparison

Strategy	Traffic Shift	Rollback Speed	Cost	Risk
Blue-Green	Instant (upstream or DNS swap)	Instant (swap back)	2x infrastructure	Low — old version ready
Rolling	Gradual (instance by instance)	Medium (redeploy previous)	1x infrastructure	Medium — mixed versions briefly
Canary	Percentage-based	Near-instant (drop to 0%)	1x + small canary fleet	Low — small blast radius

Choose based on: infrastructure cost tolerance, rollback requirements, and whether users can tolerate seeing mixed versions during deploy.

---

Blue-Green Deployment

Concept

Maintain two identical environments: blue (current live) and green (new version). Deploy to green, run smoke tests, then switch traffic from blue to green. Blue stays warm for instant rollback.

          ┌──────────┐
Users ──→ │  Nginx   │
          └─────┬────┘
                │ active_upstream.conf
         ┌──────┴──────┐
         ▼             ▼
  ┌─────────────┐  ┌─────────────┐
  │  Blue       │  │  Green      │
  │  :3000      │  │  :3001      │
  │  (old)      │  │  (new)      │
  └─────────────┘  └─────────────┘

Nginx Configuration

/etc/nginx/conf.d/upstreams.conf:

upstream blue {
    server 127.0.0.1:3000;
    keepalive 32;
}

upstream green {
    server 127.0.0.1:3001;
    keepalive 32;
}

/etc/nginx/active_upstream.conf (managed by deploy script):

# This file is symlinked/replaced atomically by deploy scripts.
# Do not edit manually.
proxy_pass http://blue;

/etc/nginx/sites-enabled/myapp.conf:

server {
    listen 80;
    server_name example.com;

    location / {
        include /etc/nginx/active_upstream.conf;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Health check configuration
        proxy_connect_timeout 5s;
        proxy_read_timeout 60s;
    }
}

Blue-Green Upstream Swap Script

/opt/scripts/blue-green-swap.sh:

#!/bin/bash
# Blue-Green upstream swap for Nginx
# Usage: blue-green-swap.sh [blue|green]
set -euo pipefail

TARGET=${1:-}
ACTIVE_CONF="/etc/nginx/active_upstream.conf"
NGINX_CONF="/etc/nginx/nginx.conf"
LOG_FILE="/var/log/deploy/blue-green.log"
HEALTH_CHECK_TIMEOUT=30

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; }

# ── Determine current and target upstream ─────────────────────────────────────
current=$(grep -oP 'http://\K[a-z]+' "$ACTIVE_CONF" 2>/dev/null || echo "blue")
if [[ -z "$TARGET" ]]; then
    TARGET=$([ "$current" = "blue" ] && echo "green" || echo "blue")
fi

if [[ "$TARGET" != "blue" && "$TARGET" != "green" ]]; then
    echo "Usage: $0 [blue|green]" >&2
    exit 1
fi

log "Switching from $current to $TARGET"

# ── Determine target port ──────────────────────────────────────────────────────
TARGET_PORT=$([ "$TARGET" = "blue" ] && echo "3000" || echo "3001")

# ── Health check target before switching ──────────────────────────────────────
log "Health checking $TARGET on port $TARGET_PORT..."
deadline=$((SECONDS + HEALTH_CHECK_TIMEOUT))
while [[ $SECONDS -lt $deadline ]]; do
    if curl -fsS --max-time 3 "http://127.0.0.1:${TARGET_PORT}/health" > /dev/null 2>&1; then
        log "$TARGET is healthy"
        break
    fi
    log "Waiting for $TARGET health check..."
    sleep 2
done

if ! curl -fsS --max-time 3 "http://127.0.0.1:${TARGET_PORT}/health" > /dev/null 2>&1; then
    log "ERROR: $TARGET health check failed after ${HEALTH_CHECK_TIMEOUT}s — aborting swap"
    exit 1
fi

# ── Write new active_upstream.conf ────────────────────────────────────────────
# Write to temp file first, then atomically move
TMP_CONF=$(mktemp)
cat > "$TMP_CONF" <<EOF
# Active upstream: $TARGET (updated $(date '+%Y-%m-%d %H:%M:%S'))
proxy_pass http://${TARGET};
EOF
sudo mv "$TMP_CONF" "$ACTIVE_CONF"

# ── Test Nginx config and reload ──────────────────────────────────────────────
if ! sudo nginx -t 2>&1; then
    log "ERROR: Nginx config test failed — reverting"
    echo "proxy_pass http://${current};" | sudo tee "$ACTIVE_CONF" > /dev/null
    exit 1
fi

sudo nginx -s reload
log "Nginx reloaded — now serving from $TARGET"

# ── Final external health check ───────────────────────────────────────────────
sleep 2
if curl -fsS --max-time 5 "https://example.com/health" > /dev/null 2>&1; then
    log "External health check passed — deploy complete"
else
    log "WARNING: External health check failed — monitor closely"
fi

echo ""
echo "Deploy summary:"
echo "  Previous: $current"
echo "  Active:   $TARGET"
echo "  Port:     $TARGET_PORT"
echo ""
echo "To rollback: sudo /opt/scripts/blue-green-swap.sh $current"

sudo chmod +x /opt/scripts/blue-green-swap.sh
sudo mkdir -p /var/log/deploy

# Deploy to green and switch
./deploy-to-green.sh    # Deploy your app to port 3001
sudo /opt/scripts/blue-green-swap.sh green

# Instant rollback
sudo /opt/scripts/blue-green-swap.sh blue

---

Rolling Deployment

Nginx upstream with max_fails for automatic routing around unhealthy instances:

upstream app {
    server 127.0.0.1:3000 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:3001 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:3002 max_fails=3 fail_timeout=30s;
    keepalive 32;
}

Rolling update loop:

#!/bin/bash
# Rolling update: restart each instance sequentially
APP_INSTANCES=(myapp@3000 myapp@3001 myapp@3002)
HEALTH_URL="http://127.0.0.1"

for INSTANCE in "${APP_INSTANCES[@]}"; do
    PORT=${INSTANCE##*@}
    echo "Restarting $INSTANCE..."
    sudo systemctl restart "$INSTANCE"

    echo "Waiting for $INSTANCE to become healthy..."
    for i in $(seq 1 30); do
        if curl -fsS --max-time 2 "${HEALTH_URL}:${PORT}/health" > /dev/null 2>&1; then
            echo "$INSTANCE is healthy after ${i}s"
            break
        fi
        sleep 1
    done

    if ! curl -fsS --max-time 2 "${HEALTH_URL}:${PORT}/health" > /dev/null 2>&1; then
        echo "ERROR: $INSTANCE failed health check — stopping rolling update"
        exit 1
    fi

    echo "$INSTANCE healthy — continuing to next instance..."
    sleep 2   # Brief pause between restarts
done

echo "Rolling update complete"

---

Canary Deployment

Route a percentage of traffic to the new version using Nginx split_clients:

# Hash on client IP for sticky routing (same user always sees same version)
split_clients "${remote_addr}" $upstream_pool {
    10%     canary;   # 10% to new version
    *       stable;   # 90% to current version
}

upstream stable {
    server 127.0.0.1:3000;
    keepalive 32;
}

upstream canary {
    server 127.0.0.1:3001;
    keepalive 32;
}

server {
    location / {
        proxy_pass http://$upstream_pool;
    }
}

Adjust percentage by editing the split_clients block and reloading Nginx. Monitor error rate and latency in Grafana before increasing canary percentage. To promote canary to 100%: replace stable backend with canary code and set * to the stable upstream.

---

Rsync + Symlink Atomic Deploy Script

Directory Layout (Capistrano-style)

/var/www/myapp/
├── releases/
│   ├── 20241201_030000/    ← older release
│   ├── 20241202_030000/    ← previous release
│   └── 20241203_030000/    ← newest release
├── current -> releases/20241203_030000   ← symlink (atomic)
└── shared/
    ├── .env                ← never in repository
    ├── logs/ -> /var/log/myapp/
    └── uploads/

Deploy Script

/opt/scripts/deploy.sh:

#!/bin/bash
# Rsync + atomic symlink deploy
# Usage: deploy.sh [--rollback]
set -euo pipefail

# ── Config ────────────────────────────────────────────────────────────────────
APP_DIR="/var/www/myapp"
RELEASES_DIR="$APP_DIR/releases"
SHARED_DIR="$APP_DIR/shared"
CURRENT_LINK="$APP_DIR/current"
SOURCE_DIR="/tmp/myapp-deploy"        # Checked-out source from CI
SERVICE_NAME="myapp"
HEALTH_URL="http://127.0.0.1:3000/health"
KEEP_RELEASES=5
LOG_FILE="/var/log/deploy/deploy.log"

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; }

mkdir -p "$RELEASES_DIR" /var/log/deploy

# ── Rollback mode ─────────────────────────────────────────────────────────────
if [[ "${1:-}" == "--rollback" ]]; then
    log "ROLLBACK: finding previous release..."
    # Get the second-most-recent release (current is most recent, previous is before that)
    PREVIOUS_RELEASE=$(ls -t "$RELEASES_DIR" | sed -n '2p')
    if [[ -z "$PREVIOUS_RELEASE" ]]; then
        log "ERROR: No previous release found"
        exit 1
    fi
    log "Rolling back to $PREVIOUS_RELEASE"
    ln -sfn "$RELEASES_DIR/$PREVIOUS_RELEASE" "$CURRENT_LINK"
    sudo systemctl restart "$SERVICE_NAME"
    sleep 3
    if curl -fsS --max-time 5 "$HEALTH_URL" > /dev/null; then
        log "Rollback successful — now running $PREVIOUS_RELEASE"
    else
        log "ERROR: Health check failed after rollback"
        exit 1
    fi
    exit 0
fi

# ── Create release directory ──────────────────────────────────────────────────
RELEASE_NAME=$(date +%Y%m%d_%H%M%S)
RELEASE_DIR="$RELEASES_DIR/$RELEASE_NAME"
mkdir -p "$RELEASE_DIR"
log "Creating release: $RELEASE_NAME"

# ── Rsync source to release directory ────────────────────────────────────────
log "Syncing files..."
rsync -av --delete \
    --exclude='.git/' \
    --exclude='node_modules/' \
    --exclude='.env*' \
    --exclude='*.log' \
    --exclude='uploads/' \
    "$SOURCE_DIR/" \
    "$RELEASE_DIR/"

# ── Link shared files and directories ────────────────────────────────────────
log "Linking shared resources..."
ln -sf "$SHARED_DIR/.env" "$RELEASE_DIR/.env"
ln -sf "$SHARED_DIR/uploads" "$RELEASE_DIR/public/uploads"
mkdir -p "$RELEASE_DIR/log"
ln -sf /var/log/myapp "$RELEASE_DIR/log"

# ── Install dependencies ──────────────────────────────────────────────────────
log "Installing dependencies..."
cd "$RELEASE_DIR"
npm ci --omit=dev --prefer-offline

# ── Run database migrations ───────────────────────────────────────────────────
log "Running database migrations..."
npm run db:migrate -- --no-rollback-on-fail
# Use additive-only migrations (expand/contract pattern for zero-downtime)

# ── Build assets (if applicable) ─────────────────────────────────────────────
if [[ -f "$RELEASE_DIR/package.json" ]] && grep -q '"build"' "$RELEASE_DIR/package.json"; then
    log "Building assets..."
    npm run build
fi

# ── Atomic symlink swap ───────────────────────────────────────────────────────
log "Swapping symlink to $RELEASE_NAME..."
ln -sfn "$RELEASE_DIR" "$CURRENT_LINK"

# ── Restart service ───────────────────────────────────────────────────────────
log "Restarting $SERVICE_NAME..."
sudo systemctl restart "$SERVICE_NAME"

# ── Health check polling ──────────────────────────────────────────────────────
log "Polling health check..."
HEALTH_TIMEOUT=60
for i in $(seq 1 $HEALTH_TIMEOUT); do
    if curl -fsS --max-time 3 "$HEALTH_URL" > /dev/null 2>&1; then
        log "Health check passed after ${i}s"
        break
    fi
    if [[ $i -eq $HEALTH_TIMEOUT ]]; then
        log "ERROR: Health check failed after ${HEALTH_TIMEOUT}s — triggering rollback"
        "$0" --rollback
        exit 1
    fi
    sleep 1
done

# ── Clean old releases ────────────────────────────────────────────────────────
log "Cleaning up old releases (keeping last $KEEP_RELEASES)..."
ls -t "$RELEASES_DIR" | tail -n +$((KEEP_RELEASES + 1)) | while read -r OLD_RELEASE; do
    log "Removing old release: $OLD_RELEASE"
    rm -rf "$RELEASES_DIR/$OLD_RELEASE"
done

log "Deploy complete: $RELEASE_NAME"
echo ""
echo "Release: $RELEASE_NAME"
echo "Current: $(readlink -f "$CURRENT_LINK")"

sudo chmod +x /opt/scripts/deploy.sh

# Deploy
sudo /opt/scripts/deploy.sh

# Rollback
sudo /opt/scripts/deploy.sh --rollback

---

GitHub Actions Deploy Workflow

.github/workflows/deploy.yml:

name: Deploy

on:
  push:
    branches: [main]
  workflow_dispatch:
    inputs:
      rollback:
        description: 'Rollback to previous release'
        type: boolean
        default: false

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - run: npm test

  deploy:
    needs: test
    runs-on: ubuntu-latest
    environment: production
    concurrency:
      group: production-deploy
      cancel-in-progress: false   # Never cancel in-flight deploys

    steps:
      - uses: actions/checkout@v4

      # Set up SSH agent with deploy key
      - name: Set up SSH
        uses: webfactory/ssh-agent@v0.8.0
        with:
          ssh-private-key: ${{ secrets.DEPLOY_SSH_KEY }}

      # Add server to known_hosts (prevents MITM prompt)
      - name: Add server to known_hosts
        run: |
          mkdir -p ~/.ssh
          ssh-keyscan -H ${{ secrets.DEPLOY_HOST }} >> ~/.ssh/known_hosts
          chmod 600 ~/.ssh/known_hosts

      # Copy source code to server (only if not rollback)
      - name: Sync source to server
        if: ${{ !inputs.rollback }}
        run: |
          rsync -av --delete \
            --exclude='.git/' \
            --exclude='node_modules/' \
            -e "ssh -o StrictHostKeyChecking=yes" \
            ./ \
            ${{ secrets.DEPLOY_USER }}@${{ secrets.DEPLOY_HOST }}:/tmp/myapp-deploy/

      # Run deploy script on server
      - name: Deploy
        run: |
          if [[ "${{ inputs.rollback }}" == "true" ]]; then
            ssh ${{ secrets.DEPLOY_USER }}@${{ secrets.DEPLOY_HOST }} \
              "sudo /opt/scripts/deploy.sh --rollback"
          else
            ssh ${{ secrets.DEPLOY_USER }}@${{ secrets.DEPLOY_HOST }} \
              "sudo /opt/scripts/deploy.sh"
          fi

      # Poll health check from GitHub Actions runner (external validation)
      - name: External health check
        run: |
          echo "Polling https://${{ secrets.DEPLOY_HOST }}/health..."
          for i in $(seq 1 30); do
            STATUS=$(curl -o /dev/null -s -w "%{http_code}" https://${{ secrets.DEPLOY_HOST }}/health)
            if [[ "$STATUS" == "200" ]]; then
              echo "Health check passed (HTTP $STATUS) after ${i}s"
              exit 0
            fi
            echo "Attempt $i: HTTP $STATUS — retrying..."
            sleep 2
          done
          echo "Health check never passed — deploy may have failed"
          exit 1

      - name: Notify Slack on failure
        if: failure()
        run: |
          curl -X POST ${{ secrets.SLACK_WEBHOOK_URL }} \
            -H 'Content-Type: application/json' \
            -d '{"text": ":red_circle: Deploy to production failed! Check GitHub Actions."}'

Required GitHub secrets: DEPLOY_SSH_KEY, DEPLOY_HOST, DEPLOY_USER, SLACK_WEBHOOK_URL.

The deploy user needs passwordless sudo for /opt/scripts/deploy.sh:

# /etc/sudoers.d/deploy
deploy ALL=(ALL) NOPASSWD: /opt/scripts/deploy.sh
deploy ALL=(ALL) NOPASSWD: /bin/systemctl restart myapp

---

Rollback Runbook

# ROLLBACK RUNBOOK — Production Deploy Rollback
# Trigger: health check failing post-deploy, error rate spike, user reports

## Step 1: Detect and Confirm Problem
  - [ ] Check Grafana: error rate, response time, 5xx count
  - [ ] Check application logs: journalctl -u myapp --since "5 minutes ago"
  - [ ] Confirm: problem started after deploy, not before

## Step 2: Immediate Mitigation (choose one)

  Option A — Symlink rollback (< 30 seconds):
    sudo /opt/scripts/deploy.sh --rollback

  Option B — Blue-green swap back (< 10 seconds):
    sudo /opt/scripts/blue-green-swap.sh blue

  Option C — Manual symlink swap:
    PREVIOUS=$(ls -t /var/www/myapp/releases | sed -n '2p')
    ln -sfn "/var/www/myapp/releases/$PREVIOUS" /var/www/myapp/current
    sudo systemctl restart myapp

## Step 3: Verify
  - [ ] curl https://example.com/health → 200 OK
  - [ ] Check Grafana: error rate returning to baseline
  - [ ] Check application logs: no new errors

## Step 4: Communicate
  - [ ] Post in #incidents: "Rolled back deploy at HH:MM — investigating root cause"
  - [ ] Update status page if customer-visible

## Step 5: Root Cause and Post-Mortem
  - [ ] Identify what caused the failure (diff the two releases)
  - [ ] Fix the issue in a branch
  - [ ] Add test coverage for the bug
  - [ ] Re-deploy after thorough testing
  - [ ] Write 5-minute post-mortem (what happened, why, what we changed)

---

Environment Management

Secrets and environment configuration must never live in the repository.

Systemd EnvironmentFile

/etc/systemd/system/myapp.service:

[Unit]
Description=My Application
After=network.target

[Service]
Type=simple
User=myapp
WorkingDirectory=/var/www/myapp/current
EnvironmentFile=/var/www/myapp/shared/.env
ExecStart=/usr/bin/node /var/www/myapp/current/dist/server.js
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

/var/www/myapp/shared/.env (chmod 640, owned by myapp:myapp):

NODE_ENV=production
DATABASE_URL=postgresql://user:pass@localhost:5432/myapp_prod
REDIS_URL=redis://localhost:6379
SECRET_KEY=very-long-random-secret

Inject from CI secrets (never commit):

# In GitHub Actions — write .env from secrets before rsync
cat > /tmp/myapp.env <<EOF
NODE_ENV=production
DATABASE_URL=${{ secrets.DATABASE_URL }}
SECRET_KEY=${{ secrets.SECRET_KEY }}
EOF
scp /tmp/myapp.env deploy@server:/var/www/myapp/shared/.env

---

Database Migration Strategy

Expand/Contract (Forward-Only) Pattern

Zero-downtime migrations require that new code and old code can run against the same database schema simultaneously.

Phase	Schema Change	Code
Expand	Add new column/table (nullable or with default)	Both old and new code work
Deploy	Roll out new code	New code uses new column
Contract	Remove old column/table (old code no longer running)	Cleanup migration

Safe live changes (additive-only):

• Adding a nullable column
• Adding a column with a DEFAULT value
• Creating a new table
• Adding an index CONCURRENTLY (Postgres): CREATE INDEX CONCURRENTLY idx_name ON table(col)

Unsafe live changes (require maintenance window or expand/contract):

• Renaming a column or table
• Dropping a column
• Adding a NOT NULL column without DEFAULT
• Adding a unique constraint (locks table)
• Changing a column type

# Postgres: create index without locking the table
psql -c "CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_users_email ON users(email);"

---

Anti-Patterns

Anti-Pattern	Problem	Fix
Deploy directly to production without staging	No validation that new version works before users see it	Always deploy to staging first; run smoke tests
No health check after deploy	Failed deploy discovered by users, not monitoring	Poll /health endpoint in deploy script; auto-rollback on failure
Migrations that break old code	During rolling deploy, old pods crash against new schema	Use expand/contract: deploy backward-compatible schema first, then new code
No rollback plan	Incident duration extends while team figures out how to revert	Test rollback on staging quarterly; keep rollback script in deploy pipeline
Storing secrets in deploy script or repository	Credential exposure in git history	Use EnvironmentFile in systemd; inject secrets from CI environment
Running database migrations in the deploy script (blocking)	Long-running migration holds deploy and may hold table lock	Run migrations separately before code deploy; use CREATE INDEX CONCURRENTLY
Single latest tag for Docker images	Previous version destroyed — rollback requires rebuild	Tag images with git SHA: myapp:abc1234; keep last N images
No concurrent deploy protection	Two deploys in parallel corrupt release directory	Use concurrency: group: production-deploy in GitHub Actions
Deploying without draining connections from old version	In-flight requests to old instance are killed mid-response	Add SIGTERM handler to app; configure systemd TimeoutStopSec=30

---

Troubleshooting

Symptom	Likely Cause	Diagnostic & Fix
Health check never passes after deploy	App crashes on startup, wrong port, or missing env var	journalctl -u myapp -n 50; check port with ss -tlnp; verify .env file readable
Symlink swap race (two deploys collide)	Concurrent deploys overwrite each other's current link	Add concurrency group in CI; use deploy lock file (flock)
SSH key not found in CI	Secret not set, or wrong secret name	Check GitHub Actions secrets tab; test with `echo "${{ secrets.DEPLOY_SSH_KEY }}"
rsync: connection unexpectedly closed	SSH auth failure or server unreachable	Test manually: ssh -i key deploy@host 'echo ok'; check known_hosts
Database migration fails mid-deploy	Migration error, constraint violation, or timeout	Roll back migration manually; restore from pre-deploy backup if data corrupted
sudo: /opt/scripts/deploy.sh: command not found	sudo PATH doesn't include script directory	Use full absolute path in sudoers; verify script exists at that path
Old release not cleaned up; disk fills up	KEEP_RELEASES cleanup not running or failing	ls -lt /var/www/myapp/releases; run cleanup manually; check deploy script exit code
Nginx returns 502 after upstream swap	New backend not ready, or upstream block misconfigured	nginx -t; check upstream port is listening (ss -tlnp); check Nginx error log
ln -sfn doesn't update symlink	Target directory doesn't exist, or path typo	ls -la /var/www/myapp/current; verify RELEASE_DIR was created successfully
Rollback deploys same broken version	Most recent and second-most-recent releases are both broken	Check release list: ls -lt releases/; manually point to an older known-good release