zero-downtime-deploy

name	zero-downtime-deploy
disable-model-invocation	true
description	Zero-downtime deployment strategies and automation: blue-green, rolling, and canary deployments with Nginx upstream switching, symlink-based atomic deploys, rsync deploy scripts, GitHub Actions SSH deploy workflows, health check polling, and rollback runbooks. USE WHEN: - Implementing blue-green deployment with Nginx upstream swapping - Writing a production deploy script with symlink-based atomic release management - Configuring canary traffic splitting with Nginx split_clients - Setting up a GitHub Actions workflow that deploys via SSH with health check polling - Writing or testing a rollback runbook - Planning database migration strategy alongside zero-downtime deploys DO NOT USE FOR: - Kubernetes rolling updates and canary with Argo Rollouts (use the kubernetes skill) - Container image building and registry management (use the docker skill) - Cloud-specific deployment (AWS CodeDeploy, GCP Cloud Deploy — use the cloud skill) - Frontend-only CDN cache invalidation strategies
allowed-tools	Read, Grep, Glob, Write, Edit, Bash

Zero-Downtime Deployment — Strategies and Automation

Strategy Comparison

Strategy	Traffic Shift	Rollback Speed	Cost	Risk
Blue-Green	Instant (upstream or DNS swap)	Instant (swap back)	2x infrastructure	Low — old version ready
Rolling	Gradual (instance by instance)	Medium (redeploy previous)	1x infrastructure	Medium — mixed versions briefly
Canary	Percentage-based	Near-instant (drop to 0%)	1x + small canary fleet	Low — small blast radius

Choose based on: infrastructure cost tolerance, rollback requirements, and whether users can tolerate seeing mixed versions during deploy.

---

Blue-Green Deployment

Concept

Maintain two identical environments: blue (current live) and green (new version). Deploy to green, run smoke tests, then switch traffic from blue to green. Blue stays warm for instant rollback.

          ┌──────────┐
Users ──→ │  Nginx   │
          └─────┬────┘
                │ active_upstream.conf
         ┌──────┴──────┐
         ▼             ▼
  ┌─────────────┐  ┌─────────────┐
  │  Blue       │  │  Green      │
  │  :3000      │  │  :3001      │
  │  (old)      │  │  (new)      │
  └─────────────┘  └─────────────┘

Nginx Configuration

/etc/nginx/conf.d/upstreams.conf:

upstream blue {
    server 127.0.0.1:3000;
    keepalive 32;
}

upstream green {
    server 127.0.0.1:3001;
    keepalive 32;
}

/etc/nginx/active_upstream.conf (managed by deploy script):

# This file is symlinked/replaced atomically by deploy scripts.
# Do not edit manually.
proxy_pass http://blue;

/etc/nginx/sites-enabled/myapp.conf:

server {
    listen 80;
    server_name example.com;

    location / {
        include /etc/nginx/active_upstream.conf;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Health check configuration
        proxy_connect_timeout 5s;
        proxy_read_timeout 60s;
    }
}

Blue-Green Upstream Swap Script

/opt/scripts/blue-green-swap.sh:

#!/bin/bash
# Blue-Green upstream swap for Nginx
# Usage: blue-green-swap.sh [blue|green]
set -euo pipefail

TARGET=${1:-}
ACTIVE_CONF="/etc/nginx/active_upstream.conf"
NGINX_CONF="/etc/nginx/nginx.conf"
LOG_FILE="/var/log/deploy/blue-green.log"
HEALTH_CHECK_TIMEOUT=30

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; }

# ── Determine current and target upstream ─────────────────────────────────────
current=$(grep -oP 'http://\K[a-z]+' "$ACTIVE_CONF" 2>/dev/null || echo "blue")
if [[ -z "$TARGET" ]]; then
    TARGET=$([ "$current" = "blue" ] && echo "green" || echo "blue")
fi

if [[ "$TARGET" != "blue" && "$TARGET" != "green" ]]; then
    echo "Usage: $0 [blue|green]" >&2
    exit 1
fi

log "Switching from $current to $TARGET"


TARGET_PORT=$([  =  ] &&   ||  )


 
deadline=$((SECONDS + HEALTH_CHECK_TIMEOUT))
 [[  -lt  ]]; 
     curl -fsS --max-time 3  > /dev/null 2>&1; 
         
        
    
     
     2


 ! curl -fsS --max-time 3  > /dev/null 2>&1; 
     
     1




TMP_CONF=$()
 >  <<
   


 !  nginx -t 2>&1; 
     
      |    > /dev/null
     1


 nginx -s reload
 


 2
 curl -fsS --max-time 5  > /dev/null 2>&1; 
     

     


 
 
 
 
 
 
 

sudo chmod +x /opt/scripts/blue-green-swap.sh
sudo mkdir -p /var/log/deploy

# Deploy to green and switch
./deploy-to-green.sh    # Deploy your app to port 3001
sudo /opt/scripts/blue-green-swap.sh green

# Instant rollback
sudo /opt/scripts/blue-green-swap.sh blue

---

Rolling Deployment

Nginx upstream with max_fails for automatic routing around unhealthy instances:

upstream app {
    server 127.0.0.1:3000 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:3001 max_fails=3 fail_timeout=30s;
    server 127.0.0.1:3002 max_fails=3 fail_timeout=30s;
    keepalive 32;
}

Rolling update loop:

#!/bin/bash
# Rolling update: restart each instance sequentially
APP_INSTANCES=(myapp@3000 myapp@3001 myapp@3002)
HEALTH_URL="http://127.0.0.1"

for INSTANCE in "${APP_INSTANCES[@]}"; do
    PORT=${INSTANCE##*@}
    echo "Restarting $INSTANCE..."
    sudo systemctl restart "$INSTANCE"

    echo "Waiting for $INSTANCE to become healthy..."
    for i in $(seq 1 30); do
        if curl -fsS --max-time 2 "${HEALTH_URL}:${PORT}/health" > /dev/null 2>&1; then
            echo "$INSTANCE is healthy after ${i}s"
            break
        fi
        sleep 1
    done

    if ! curl -fsS --max-time 2 "${HEALTH_URL}:${PORT}/health" > /dev/null 2>&1; then
        echo "ERROR: $INSTANCE failed health check — stopping rolling update"
        exit 1
    fi

    echo "$INSTANCE healthy — continuing to next instance..."
    sleep 2   


 

---

Canary Deployment

Route a percentage of traffic to the new version using Nginx split_clients:

# Hash on client IP for sticky routing (same user always sees same version)
split_clients "${remote_addr}" $upstream_pool {
    10%     canary;   # 10% to new version
    *       stable;   # 90% to current version
}

upstream stable {
    server 127.0.0.1:3000;
    keepalive 32;
}

upstream canary {
    server 127.0.0.1:3001;
    keepalive 32;
}

server {
    location / {
        proxy_pass http://$upstream_pool;
    }
}

Adjust percentage by editing the split_clients block and reloading Nginx. Monitor error rate and latency in Grafana before increasing canary percentage. To promote canary to 100%: replace stable backend with canary code and set * to the stable upstream.

---

Rsync + Symlink Atomic Deploy Script

Directory Layout (Capistrano-style)

/var/www/myapp/
├── releases/
│   ├── 20241201_030000/    ← older release
│   ├── 20241202_030000/    ← previous release
│   └── 20241203_030000/    ← newest release
├── current -> releases/20241203_030000   ← symlink (atomic)
└── shared/
    ├── .env                ← never in repository
    ├── logs/ -> /var/log/myapp/
    └── uploads/

Deploy Script

/opt/scripts/deploy.sh:

#!/bin/bash
# Rsync + atomic symlink deploy
# Usage: deploy.sh [--rollback]
set -euo pipefail

# ── Config ────────────────────────────────────────────────────────────────────
APP_DIR="/var/www/myapp"
RELEASES_DIR="$APP_DIR/releases"
SHARED_DIR="$APP_DIR/shared"
CURRENT_LINK="$APP_DIR/current"
SOURCE_DIR="/tmp/myapp-deploy"        # Checked-out source from CI
SERVICE_NAME="myapp"
HEALTH_URL="http://127.0.0.1:3000/health"
KEEP_RELEASES=5
LOG_FILE="/var/log/deploy/deploy.log"

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; }

mkdir -p "$RELEASES_DIR" /var/log/deploy

# ── Rollback mode ─────────────────────────────────────────────────────────────
if [[ "${1:-}" == "--rollback" ]]; then
    log "ROLLBACK: finding previous release..."
    # Get the second-most-recent release (current is most recent, previous is before that)
    PREVIOUS_RELEASE=$(ls -t "$RELEASES_DIR" | sed -n '2p')
    if [[ -z "$PREVIOUS_RELEASE" ]]; then
        log "ERROR: No previous release found"
         1
    
     
     -sfn  
     systemctl restart 
     3
     curl -fsS --max-time 5  > /dev/null; 
         
    
         
         1
    
     0



RELEASE_NAME=$( +%Y%m%d_%H%M%S)
RELEASE_DIR=
 -p 
 


 
rsync -av --delete \
    --exclude= \
    --exclude= \
    --exclude= \
    --exclude= \
    --exclude= \
     \
    


 
 -sf  
 -sf  
 -p 
 -sf /var/log/myapp 


 
 
npm ci --omit=dev --prefer-offline


 
npm run db:migrate -- --no-rollback-on-fail



 [[ -f  ]] && grep -q  ; 
     
    npm run build



 
 -sfn  


 
 systemctl restart 


 
HEALTH_TIMEOUT=60
 i  $( 1 ); 
     curl -fsS --max-time 3  > /dev/null 2>&1; 
         
        
    
     [[  -eq  ]]; 
         
         --rollback
         1
    
     1



 
 -t  |  -n +$((KEEP_RELEASES + )) |   -r OLD_RELEASE; 
     
     -rf 


 
 
 
 

sudo chmod +x /opt/scripts/deploy.sh

# Deploy
sudo /opt/scripts/deploy.sh

# Rollback
sudo /opt/scripts/deploy.sh --rollback

---

GitHub Actions Deploy Workflow

.github/workflows/deploy.yml:

name: Deploy

on:
  push:
    branches: [main]
  workflow_dispatch:
    inputs:
      rollback:
        description: 'Rollback to previous release'
        type: boolean
        default: false

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm
      - run: npm ci
      - run: npm test

  deploy:
    needs: test
    runs-on: ubuntu-latest
    environment: production
    concurrency:
      group: production-deploy
      cancel-in-progress: false   # Never cancel in-flight deploys

    steps:
      - uses: 

      
          
         
        
             

      
           
         
      
           
           
         
      
        
         
      
          
         
           
         
         

Required GitHub secrets: DEPLOY_SSH_KEY, DEPLOY_HOST, DEPLOY_USER, SLACK_WEBHOOK_URL.

The deploy user needs passwordless sudo for /opt/scripts/deploy.sh:

# /etc/sudoers.d/deploy
deploy ALL=(ALL) NOPASSWD: /opt/scripts/deploy.sh
deploy ALL=(ALL) NOPASSWD: /bin/systemctl restart myapp

---

Rollback Runbook

# ROLLBACK RUNBOOK — Production Deploy Rollback
# Trigger: health check failing post-deploy, error rate spike, user reports

## Step 1: Detect and Confirm Problem
  - [ ] Check Grafana: error rate, response time, 5xx count
  - [ ] Check application logs: journalctl -u myapp --since "5 minutes ago"
  - [ ] Confirm: problem started after deploy, not before

## Step 2: Immediate Mitigation (choose one)

  Option A — Symlink rollback (< 30 seconds):
    sudo /opt/scripts/deploy.sh --rollback

  Option B — Blue-green swap back (< 10 seconds):
    sudo /opt/scripts/blue-green-swap.sh blue

  Option C — Manual symlink swap:
    PREVIOUS=$(ls -t /var/www/myapp/releases | sed -n '2p')
    ln -sfn "/var/www/myapp/releases/$PREVIOUS" /var/www/myapp/current
    sudo systemctl restart myapp

## Step 3: Verify
  - [ ] curl https://example.com/health → 200 OK
  - [ ] Check Grafana: error rate returning to baseline
  - [ ] Check application logs: no new errors

## Step 4: Communicate
  - [ ] Post in #incidents: "Rolled back deploy at HH:MM — investigating root cause"
  - [ ] Update status page if customer-visible

## Step 5: Root Cause and Post-Mortem
  - [ ] Identify what caused the failure (diff the two releases)
  - [ ] Fix the issue in a branch
  - [ ] Add test coverage for the bug
  - [ ] Re-deploy after thorough testing
  - [ ] Write 5-minute post-mortem (what happened, why, what we changed)

---

Environment Management

Secrets and environment configuration must never live in the repository.

Systemd EnvironmentFile

/etc/systemd/system/myapp.service:

[Unit]
Description=My Application
After=network.target

[Service]
Type=simple
User=myapp
WorkingDirectory=/var/www/myapp/current
EnvironmentFile=/var/www/myapp/shared/.env
ExecStart=/usr/bin/node /var/www/myapp/current/dist/server.js
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

/var/www/myapp/shared/.env (chmod 640, owned by myapp:myapp):

NODE_ENV=production
DATABASE_URL=postgresql://user:pass@localhost:5432/myapp_prod
REDIS_URL=redis://localhost:6379
SECRET_KEY=very-long-random-secret

Inject from CI secrets (never commit):

# In GitHub Actions — write .env from secrets before rsync
cat > /tmp/myapp.env <<EOF
NODE_ENV=production
DATABASE_URL=${{ secrets.DATABASE_URL }}
SECRET_KEY=${{ secrets.SECRET_KEY }}
EOF
scp /tmp/myapp.env deploy@server:/var/www/myapp/shared/.env

---

Database Migration Strategy

Expand/Contract (Forward-Only) Pattern

Zero-downtime migrations require that new code and old code can run against the same database schema simultaneously.

Phase	Schema Change	Code
Expand	Add new column/table (nullable or with default)	Both old and new code work
Deploy	Roll out new code	New code uses new column
Contract	Remove old column/table (old code no longer running)	Cleanup migration

Safe live changes (additive-only):

• Adding a nullable column
• Adding a column with a DEFAULT value
• Creating a new table
• Adding an index CONCURRENTLY (Postgres): CREATE INDEX CONCURRENTLY idx_name ON table(col)

Unsafe live changes (require maintenance window or expand/contract):

• Renaming a column or table
• Dropping a column
• Adding a NOT NULL column without DEFAULT
• Adding a unique constraint (locks table)
• Changing a column type

# Postgres: create index without locking the table
psql -c "CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_users_email ON users(email);"

---

Anti-Patterns

Anti-Pattern	Problem	Fix
Deploy directly to production without staging	No validation that new version works before users see it	Always deploy to staging first; run smoke tests
No health check after deploy	Failed deploy discovered by users, not monitoring	Poll /health endpoint in deploy script; auto-rollback on failure
Migrations that break old code	During rolling deploy, old pods crash against new schema	Use expand/contract: deploy backward-compatible schema first, then new code
No rollback plan	Incident duration extends while team figures out how to revert	Test rollback on staging quarterly; keep rollback script in deploy pipeline
Storing secrets in deploy script or repository	Credential exposure in git history	Use EnvironmentFile in systemd; inject secrets from CI environment
Running database migrations in the deploy script (blocking)	Long-running migration holds deploy and may hold table lock	Run migrations separately before code deploy; use CREATE INDEX CONCURRENTLY
Single latest tag for Docker images	Previous version destroyed — rollback requires rebuild	Tag images with git SHA: myapp:abc1234; keep last N images
No concurrent deploy protection	Two deploys in parallel corrupt release directory	Use concurrency: group: production-deploy in GitHub Actions
Deploying without draining connections from old version	In-flight requests to old instance are killed mid-response	Add SIGTERM handler to app; configure systemd TimeoutStopSec=30

---

Troubleshooting

Symptom	Likely Cause	Diagnostic & Fix
Health check never passes after deploy	App crashes on startup, wrong port, or missing env var	journalctl -u myapp -n 50; check port with ss -tlnp; verify .env file readable
Symlink swap race (two deploys collide)	Concurrent deploys overwrite each other's current link	Add concurrency group in CI; use deploy lock file (flock)
SSH key not found in CI	Secret not set, or wrong secret name	Check GitHub Actions secrets tab; test with `echo "${{ secrets.DEPLOY_SSH_KEY }}"
rsync: connection unexpectedly closed	SSH auth failure or server unreachable	Test manually: ssh -i key deploy@host 'echo ok'; check known_hosts
Database migration fails mid-deploy	Migration error, constraint violation, or timeout	Roll back migration manually; restore from pre-deploy backup if data corrupted
sudo: /opt/scripts/deploy.sh: command not found	sudo PATH doesn't include script directory	Use full absolute path in sudoers; verify script exists at that path
Old release not cleaned up; disk fills up	KEEP_RELEASES cleanup not running or failing	ls -lt /var/www/myapp/releases; run cleanup manually; check deploy script exit code
Nginx returns 502 after upstream swap	New backend not ready, or upstream block misconfigured	nginx -t; check upstream port is listening (ss -tlnp); check Nginx error log
ln -sfn doesn't update symlink	Target directory doesn't exist, or path typo	ls -la /var/www/myapp/current; verify RELEASE_DIR was created successfully
Rollback deploys same broken version	Most recent and second-most-recent releases are both broken	Check release list: ls -lt releases/; manually point to an older known-good release