production-deploy

name

production-deploy

description

Pre-deployment validation and release management — structured checklists for database migrations, environment variables, rollback plans, backward compatibility, and deployment strategies. Use this skill when the user mentions deploy, release, ship to prod, merge to main, CI/CD pipeline, or says /production deploy. Triggers on deployment-related discussions, release planning, or pre-release validation.

Production Deploy

This skill encodes the pre-deployment discipline that separates teams who ship confidently from teams who ship and pray. Every checklist item here exists because someone skipped it and caused an outage. The patterns are opinionated and battle-tested — this is not a deployment tutorial, it is a deployment gate.

If you cannot check every box, you are not ready to deploy. Partial deploys are how you get partial outages.

---

1. Pre-Deployment Checklist

Run through this before every production deploy. No exceptions, no shortcuts, no "we'll check it in staging." Print this out and check the boxes with a pen if that is what it takes.

Data Layer

• All migrations reviewed and classified (see Section 2)
• Rollback migration exists and has been tested against staging
• Database backup taken (or continuous WAL archiving confirmed active)
• Migrations tested against production-volume data on staging
• No migrations hold locks longer than 2 seconds (SET lock_timeout = '2s')
• Migration order is correct (dependencies between migrations verified)

Application Layer

• All new environment variables documented and set in production (see Section 3)
• Secrets rotated if required (API keys, tokens, certificates)
• Feature flags configured for any gradual rollout
• API changes are backward-compatible (see Section 6)
• No breaking changes to event schemas, message formats, or shared contracts
• Application starts successfully with production config on staging

Infrastructure

• Health check endpoints respond correctly (/health/live, /health/ready)
• Graceful shutdown tested — in-flight requests complete before process exits
• Resource limits set (CPU, memory) — no unbounded containers
• Autoscaling configured and tested (min/max instances, scale-up triggers)
• DNS/load balancer changes propagated (if applicable)
• TLS certificates valid and not expiring within 30 days

Observability

• Logs reaching the aggregator (CloudWatch, Datadog, Grafana Loki)
• Key metrics have alerts: error rate > threshold, latency p99 > SLA, pod restarts
• Distributed traces enabled and sampling rate appropriate for production
• Error budget reviewed — are we within SLO? If not, this deploy needs extra scrutiny
• Dashboard updated with new metrics if the deploy adds new endpoints or features

Rollback Readiness

• Rollback plan written and reviewed (see Section 5)
• Previous deployment artifact is available and verified (image tag, release SHA)
• Team knows who has deploy access and who is on-call
• Incident channel identified (Slack channel, PagerDuty service, phone tree)
• Estimated time to rollback documented (should be under 5 minutes)

Detection — find deploys that skip the checklist:

# PRs merged to main without a deploy checklist comment
gh pr list --state merged --base main --limit 20 --json title,body \
  | jq '.[] | select(.body | test("deploy checklist|pre-deploy|rollback plan"; "i") | not) | .title'

# Recent deploys without associated rollback tags
git tag -l "rollback-*" --sort=-creatordate | head -5

---

2. Migration Safety Review

Every migration must be classified before it ships. This is the single most important step in any deploy that touches the database. Get it wrong and you take the service down for every user, not just the ones using the new feature.

Classification

Additive (Safe) — Green Light

These migrations are safe for zero-downtime rolling deploys. Old code ignores the new structures.

• New tables
• New nullable columns (without defaults on large tables)
• New indexes (with CONCURRENTLY)
• New views
• New functions/procedures

-- Safe: new table, no impact on existing code
CREATE TABLE notifications (
    id BIGSERIAL PRIMARY KEY,
    user_id BIGINT NOT NULL REFERENCES users(id),
    message TEXT NOT NULL,
    read_at TIMESTAMPTZ,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Safe: new nullable column, old code ignores it
ALTER TABLE orders ADD COLUMN tracking_url TEXT;

-- Safe: concurrent index, no write locks
CREATE INDEX CONCURRENTLY ix_notifications_user_id ON notifications (user_id);

Transformative (Risky) — Yellow Light

These require extra review, staging testing with production-volume data, and explicit sign-off. They can be done safely but the safe pattern is non-obvious.

• Column type changes (expand-contract)
• Data backfills on large tables (batch, not single UPDATE)
• Adding NOT NULL constraints (use NOT VALID + VALIDATE)
• Adding CHECK constraints on existing data
• Adding foreign keys on existing data

# RISKY: backfill must be batched to avoid locking the table
# and overwhelming the database with a single massive transaction

# WRONG — single UPDATE locks the table and generates huge WAL
# op.execute("UPDATE orders SET status = 'active' WHERE status IS NULL")

# RIGHT — batch in chunks
def upgrade():
    conn = op.get_bind()
    batch_size = 1000
    while True:
        result = conn.execute(text(
            "UPDATE orders SET status = 'active' "
            "WHERE id IN ("
            "  SELECT id FROM orders WHERE status IS NULL LIMIT :batch"
            ")"
        ), {"batch": batch_size})
        if result.rowcount == 0:
            break
        conn.execute(text("COMMIT"))

Destructive (Dangerous) — Red Light

REFUSE to deploy destructive migrations without explicit confirmation from the user. These are irreversible. Suggest a zero-downtime alternative first.

• Column drops
• Table drops
• Data deletion (DELETE/TRUNCATE)
• Column renames (breaks old code during rolling deploy)
• Type changes that narrow data (e.g., TEXT to VARCHAR(50))

# DANGEROUS: dropping a column breaks old code during rolling deploy
# Old instances still query this column. They crash.

# Instead of:
# op.drop_column('users', 'legacy_role')

# Use expand-contract (3 deploys):
# Deploy 1: Stop reading from legacy_role in code
# Deploy 2: Stop writing to legacy_role in code
# Deploy 3: Drop the column (only after ALL instances run new code)

# Generate rollback script BEFORE executing:
def downgrade():
    op.add_column('users', sa.Column('legacy_role', sa.String(50)))
    # NOTE: data is gone. Restore from backup if needed.

Detection — find risky migrations before they ship:

# Scan migration files for dangerous operations
grep -rn "drop_column\|drop_table\|DROP TABLE\|DROP COLUMN\|TRUNCATE\|DELETE FROM" \
  alembic/versions/ migrations/

# Scan for missing lock_timeout
grep -rL "lock_timeout" alembic/versions/*.py

# Use squawk for automated migration linting (PostgreSQL)
# pip install squawk-cli
squawk alembic/versions/latest_migration.sql

Migration Lint with squawk

squawk catches unsafe migration patterns automatically. Add it to CI:

# GitHub Actions
- name: Lint migrations
  run: |
    pip install squawk-cli
    # Generate SQL from Alembic migrations
    alembic upgrade head --sql > migration.sql
    squawk migration.sql

squawk catches: missing CONCURRENTLY on indexes, NOT NULL additions without defaults, missing lock_timeout, and more.

---

3. Environment Variable Validation

Missing environment variables are the #2 cause of deploy failures (after bad migrations). The fix is simple: validate everything at startup, fail fast with a clear error message.

Python — Pydantic BaseSettings

from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic import Field, field_validator
from typing import Annotated


class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=".env",
        env_file_encoding="utf-8",
        case_sensitive=False,
        extra="ignore",
    )

    # Required — app crashes at import time if missing
    database_url: str
    secret_key: str
    allowed_hosts: list[str]

    # Required with validation
    environment: str
    log_level: str = "INFO"

    # Optional with safe defaults
    redis_url: str = "redis://localhost:6379/0"
    cors_origins: list[str] = ["http://localhost:3000"]
    db_pool_size: int = 10
    db_max_overflow: int = 5
    sentry_dsn: str | None = None

    # Deploy metadata — set by CI/CD
    app_version: str = "dev"
    deploy_sha: str = "unknown"

    @field_validator("environment")
    @classmethod
    def validate_environment(cls, v: str) -> str:
        allowed = {"development", "staging", "production"}
        if v not in allowed:
            raise ValueError(f"environment must be one of {allowed}, got '{v}'")
        return v

    @field_validator("log_level")
    @classmethod
    def validate_log_level(cls, v: str) -> str:
        allowed = {"DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"}
        if v.upper() not in allowed:
            raise ValueError(f"log_level must be one of {allowed}, got '{v}'")
        return v.upper()


# Instantiate at module level — fails at import time, not at first request
settings = Settings()

Node.js — envalid

import { cleanEnv, str, num, url, bool } from "envalid";

const env = cleanEnv(process.env, {
  // Required — process exits with clear error if missing
  DATABASE_URL: url(),
  SECRET_KEY: str({ desc: "JWT signing key" }),
  ALLOWED_HOSTS: str({ desc: "Comma-separated allowed hosts" }),

  // Required with validation
  NODE_ENV: str({ choices: ["development", "staging", "production"] }),
  LOG_LEVEL: str({
    choices: ["debug", "info", "warn", "error"],
    default: "info",
  }),

  // Optional with safe defaults
  REDIS_URL: url({ default: "redis://localhost:6379/0" }),
  PORT: num({ default: 3000 }),
  SENTRY_DSN: str({ default: "" }),

  // Deploy metadata
  APP_VERSION: str({ default: "dev" }),
  DEPLOY_SHA: str({ default: "unknown" }),
});

export default env;

Rules

• Every required env var must be validated at startup, not on first use
• Fail with a human-readable error: "Missing required environment variable: DATABASE_URL" — not a cryptic NoneType error 3 stack frames deep
• Document every env var in a .env.example file committed to the repo
• New env vars added in a PR MUST be set in production BEFORE the deploy, or have a safe default
• Never use os.getenv("SECRET") without a fallback or validation — it silently returns None

Detection — find unvalidated env vars:

# Python: find raw os.getenv/os.environ without Settings class
grep -rn "os\.getenv\|os\.environ\[" --include="*.py" src/ app/ \
  | grep -v "settings\|config\|test"

# Node.js: find raw process.env without envalid
grep -rn "process\.env\." --include="*.ts" --include="*.js" src/ \
  | grep -v "node_modules\|config\|env\.ts\|env\.js\|test"

# Find env vars referenced in code but missing from .env.example
comm -23 \
  <(grep -rhoP '(?:os\.getenv|os\.environ\[|process\.env\.)["'"'"']?\K[A-Z_]+' src/ | sort -u) \
  <(grep -oP '^[A-Z_]+' .env.example 2>/dev/null | sort -u)

---

4. Deployment Strategies

Choose the right strategy for the risk level. There is no universal best — each has tradeoffs.

Rolling Update (Default for Stateless Services)

New instances start, pass health checks, and begin receiving traffic. Old instances drain and shut down. At any point during the deploy, both old and new code are running simultaneously.

# Kubernetes Deployment
apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # At most 1 extra pod during rollout
      maxUnavailable: 0  # Never reduce below desired count
  template:
    spec:
      containers:
        - name: api
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8000
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health/live
              port: 8000
            initialDelaySeconds: 15
            periodSeconds: 20
      terminationGracePeriodSeconds: 30

When to use: Standard deploys of stateless services with backward-compatible changes. Most deploys.

Risk: Old and new code run simultaneously. Database schema and API contracts must be backward-compatible.

Blue-Green (Zero-Downtime with Instant Rollback)

Run two identical environments. Deploy to the inactive one ("green"), verify, then switch traffic. Rollback is instant — switch traffic back to "blue."

# Cloud Run example — deploy to a new revision without serving traffic
gcloud run deploy my-service \
  --image gcr.io/my-project/my-service:${NEW_SHA} \
  --no-traffic \
  --tag canary

# Smoke test the new revision
curl -s https://canary---my-service-xxxxx.a.run.app/health/ready

# If healthy, shift all traffic
gcloud run services update-traffic my-service --to-latest

# If broken, rollback to previous revision
gcloud run services update-traffic my-service \
  --to-revisions=my-service-00042-abc=100

# Docker Compose blue-green (simplified)
# nginx.conf switches upstream between blue and green
services:
  blue:
    image: myapp:current
    ports: ["8001:8000"]
  green:
    image: myapp:${NEW_TAG}
    ports: ["8002:8000"]
  nginx:
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
    ports: ["80:80"]

When to use: Critical services where rollback speed matters more than resource cost. Services with hard availability SLAs.

Cost: Requires 2x infrastructure during deploy window. Both environments must be fully functional.

Canary (High-Risk Changes)

Route a small percentage of traffic to the new version. Monitor error rates and latency. Gradually increase traffic if metrics are healthy.

# Kubernetes canary with Istio VirtualService
apiVersion: networking.istio.io/v1
kind: VirtualService
spec:
  hosts: ["my-service"]
  http:
    - route:
        - destination:
            host: my-service
            subset: stable
          weight: 95
        - destination:
            host: my-service
            subset: canary
          weight: 5

# Cloud Run canary — send 5% of traffic to new revision
gcloud run services update-traffic my-service \
  --to-revisions=my-service-00043-def=5,my-service-00042-abc=95

# Monitor for 15 minutes, then promote or rollback
# Check error rate:
# - If canary error rate > 2x stable error rate, rollback immediately
# - If canary latency p99 > 1.5x stable p99, rollback immediately

When to use: Large schema changes, major refactors, new integrations, anything where "it worked in staging" is not sufficient confidence.

Canary promotion criteria:

1. Error rate within 1.5x of stable baseline for 15 minutes
2. Latency p99 within 1.5x of stable baseline
3. No new error types in logs
4. No increase in pod restarts

Feature Flags (Gradual Rollout at Application Level)

Decouple deploy from release. Ship the code, then enable the feature gradually. This is the safest approach for user-facing changes.

# Python — LaunchDarkly / Unleash / simple feature flag
import structlog

logger = structlog.get_logger()


def get_recommendations(user_id: str, feature_flags: FeatureFlags) -> list:
    if feature_flags.is_enabled("new_recommendation_engine", user_id=user_id):