production-deploy

# Production Deploy This skill encodes the pre-deployment discipline that separates teams who ship confidently from teams who ship and pray. Every checklist item here exists because someone skipped it and caused an outage. The patterns are opinionated and battle-tested — this is not a deployment tutorial, it is a deployment gate. If you cannot check every box, you are not ready to deploy. Partial deploys are how you get partial outages. --- ## 1. Pre-Deployment Checklist Run through this before every production deploy. No exceptions, no shortcuts, no "we'll check it in staging." Print this out and check the boxes with a pen if that is what it takes. ### Data Layer - [ ] All migrations reviewed and classified (see Section 2) - [ ] Rollback migration exists and has been tested against staging - [ ] Database backup taken (or continuous WAL archiving confirmed active) - [ ] Migrations tested against production-volume data on staging - [ ] No migrations hold locks longer than 2 seconds (`SET lock_timeout = '2s'`) - [ ] Migration order is correct (dependencies between migrations verified) ### Application Layer - [ ] All new environment variables documented and set in production (see Section 3) - [ ] Secrets rotated if required (API keys, tokens, certificates) - [ ] Feature flags configured for any gradual rollout - [ ] API changes are backward-compatible (see Section 6) - [ ] No breaking changes to event schemas, message formats, or shared contracts - [ ] Application starts successfully with production config on staging ### Infrastructure - [ ] Health check endpoints respond correctly (`/health/live`, `/health/ready`) - [ ] Graceful shutdown tested — in-flight requests complete before process exits - [ ] Resource limits set (CPU, memory) — no unbounded containers - [ ] Autoscaling configured and tested (min/max instances, scale-up triggers) - [ ] DNS/load balancer changes propagated (if applicable) - [ ] TLS certificates valid and not expiring within 30 days ### Observability - [ ] Logs reaching the aggregator (CloudWatch, Datadog, Grafana Loki) - [ ] Key metrics have alerts: error rate > threshold, latency p99 > SLA, pod restarts - [ ] Distributed traces enabled and sampling rate appropriate for production - [ ] Error budget reviewed — are we within SLO? If not, this deploy needs extra scrutiny - [ ] Dashboard updated with new metrics if the deploy adds new endpoints or features ### Rollback Readiness - [ ] Rollback plan written and reviewed (see Section 5) - [ ] Previous deployment artifact is available and verified (image tag, release SHA) - [ ] Team knows who has deploy access and who is on-call - [ ] Incident channel identified (Slack channel, PagerDuty service, phone tree) - [ ] Estimated time to rollback documented (should be under 5 minutes) **Detection — find deploys that skip the checklist:** ```bash # PRs merged to main without a deploy checklist comment gh pr list --state merged --base main --limit 20 --json title,body \ | jq '.[] | select(.body | test("deploy checklist|pre-deploy|rollback plan"; "i") | not) | .title' # Recent deploys without associated rollback tags git tag -l "rollback-*" --sort=-creatordate | head -5 ``` --- ## 2. Migration Safety Review Every migration must be classified before it ships. This is the single most important step in any deploy that touches the database. Get it wrong and you take the service down for every user, not just the ones using the new feature. ### Classification #### Additive (Safe) — Green Light These migrations are safe for zero-downtime rolling deploys. Old code ignores the new structures. - New tables - New nullable columns (without defaults on large tables) - New indexes (with `CONCURRENTLY`) - New views - New functions/procedures ```sql -- Safe: new table, no impact on existing code CREATE TABLE notifications ( id BIGSERIAL PRIMARY KEY, user_id BIGINT NOT NULL REFERENCES users(id), message TEXT NOT NULL, read_at TIMESTAMPTZ, created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); -- Safe: new nullable column, old code ignores it ALTER TABLE orders ADD COLUMN tracking_url TEXT; -- Safe: concurrent index, no write locks CREATE INDEX CONCURRENTLY ix_notifications_user_id ON notifications (user_id); ``` #### Transformative (Risky) — Yellow Light These require extra review, staging testing with production-volume data, and explicit sign-off. They can be done safely but the safe pattern is non-obvious. - Column type changes (expand-contract) - Data backfills on large tables (batch, not single UPDATE) - Adding NOT NULL constraints (use NOT VALID + VALIDATE) - Adding CHECK constraints on existing data - Adding foreign keys on existing data ```python # RISKY: backfill must be batched to avoid locking the table # and overwhelming the database with a single massive transaction # WRONG — single UPDATE locks the table and generates huge WAL # op.execute("UPDATE orders SET status = 'active' WHERE status IS NULL") # RIGHT — batch in chunks def upgrade(): conn = op.get_bind() batch_size = 1000 while True: result = conn.execute(text( "UPDATE orders SET status = 'active' " "WHERE id IN (" " SELECT id FROM orders WHERE status IS NULL LIMIT :batch" ")" ), {"batch": batch_size}) if result.rowcount == 0: break conn.execute(text("COMMIT")) ``` #### Destructive (Dangerous) — Red Light **REFUSE to deploy destructive migrations without explicit confirmation from the user.** These are irreversible. Suggest a zero-downtime alternative first. - Column drops - Table drops - Data deletion (DELETE/TRUNCATE) - Column renames (breaks old code during rolling deploy) - Type changes that narrow data (e.g., TEXT to VARCHAR(50)) ```python # DANGEROUS: dropping a column breaks old code during rolling deploy # Old instances still query this column. They crash. # Instead of: # op.drop_column('users', 'legacy_role') # Use expand-contract (3 deploys): # Deploy 1: Stop reading from legacy_role in code # Deploy 2: Stop writing to legacy_role in code # Deploy 3: Drop the column (only after ALL instances run new code) # Generate rollback script BEFORE executing: def downgrade(): op.add_column('users', sa.Column('legacy_role', sa.String(50))) # NOTE: data is gone. Restore from backup if needed. ``` **Detection — find risky migrations before they ship:** ```bash # Scan migration files for dangerous operations grep -rn "drop_column\|drop_table\|DROP TABLE\|DROP COLUMN\|TRUNCATE\|DELETE FROM" \ alembic/versions/ migrations/ # Scan for missing lock_timeout grep -rL "lock_timeout" alembic/versions/*.py # Use squawk for automated migration linting (PostgreSQL) # pip install squawk-cli squawk alembic/versions/latest_migration.sql ``` ### Migration Lint with squawk [squawk](https://squawkhq.com/) catches unsafe migration patterns automatically. Add it to CI: ```yaml # GitHub Actions - name: Lint migrations run: | pip install squawk-cli # Generate SQL from Alembic migrations alembic upgrade head --sql > migration.sql squawk migration.sql ``` squawk catches: missing `CONCURRENTLY` on indexes, `NOT NULL` additions without defaults, missing `lock_timeout`, and more. --- ## 3. Environment Variable Validation Missing environment variables are the #2 cause of deploy failures (after bad migrations). The fix is simple: validate everything at startup, fail fast with a clear error message. ### Python — Pydantic BaseSettings ```python from pydantic_settings import BaseSettings, SettingsConfigDict from pydantic import Field, field_validator from typing import Annotated class Settings(BaseSettings): model_config = SettingsConfigDict( env_file=".env", env_file_encoding="utf-8", case_sensitive=False, extra="ignore", ) # Required — app crashes at import time if missing database_url: str secret_key: str allowed_hosts: list[str] # Required with validation environment: str log_level: str = "INFO" # Optional with safe defaults redis_url: str = "redis://localhost:6379/0" cors_origins: list[str] = ["http://localhost:3000"] db_pool_size: int = 10 db_max_overflow: int = 5 sentry_dsn: str | None = None # Deploy metadata — set by CI/CD app_version: str = "dev" deploy_sha: str = "unknown" @field_validator("environment") @classmethod def validate_environment(cls, v: str) -> str: allowed = {"development", "staging", "production"} if v not in allowed: raise ValueError(f"environment must be one of {allowed}, got '{v}'") return v @field_validator("log_level") @classmethod def validate_log_level(cls, v: str) -> str: allowed = {"DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"} if v.upper() not in allowed: raise ValueError(f"log_level must be one of {allowed}, got '{v}'") return v.upper() # Instantiate at module level — fails at import time, not at first request settings = Settings() ``` ### Node.js — envalid ```typescript import { cleanEnv, str, num, url, bool } from "envalid"; const env = cleanEnv(process.env, { // Required — process exits with clear error if missing DATABASE_URL: url(), SECRET_KEY: str({ desc: "JWT signing key" }), ALLOWED_HOSTS: str({ desc: "Comma-separated allowed hosts" }), // Required with validation NODE_ENV: str({ choices: ["development", "staging", "production"] }), LOG_LEVEL: str({ choices: ["debug", "info", "warn", "error"], default: "info", }), // Optional with safe defaults REDIS_URL: url({ default: "redis://localhost:6379/0" }), PORT: num({ default: 3000 }), SENTRY_DSN: str({ default: "" }), // Deploy metadata APP_VERSION: str({ default: "dev" }), DEPLOY_SHA: str({ default: "unknown" }), }); export default env; ``` ### Rules - Every required env var must be validated at startup, not on first use - Fail with a human-readable error: "Missing required environment variable: DATABASE_URL" — not a cryptic NoneType error 3 stack frames deep - Document every env var in a `.env.example` file committed to the repo - New env vars added in a PR MUST be set in production BEFORE the deploy, or have a safe default - Never use `os.getenv("SECRET")` without a fallback or validation — it silently returns None **Detection — find unvalidated env vars:** ```bash # Python: find raw os.getenv/os.environ without Settings class grep -rn "os\.getenv\|os\.environ\[" --include="*.py" src/ app/ \ | grep -v "settings\|config\|test" # Node.js: find raw process.env without envalid grep -rn "process\.env\." --include="*.ts" --include="*.js" src/ \ | grep -v "node_modules\|config\|env\.ts\|env\.js\|test" # Find env vars referenced in code but missing from .env.example comm -23 \ <(grep -rhoP '(?:os\.getenv|os\.environ\[|process\.env\.)["'"'"']?\K[A-Z_]+' src/ | sort -u) \ <(grep -oP '^[A-Z_]+' .env.example 2>/dev/null | sort -u) ``` --- ## 4. Deployment Strategies Choose the right strategy for the risk level. There is no universal best — each has tradeoffs. ### Rolling Update (Default for Stateless Services) New instances start, pass health checks, and begin receiving traffic. Old instances drain and shut down. At any point during the deploy, both old and new code are running simultaneously. ```yaml # Kubernetes Deployment apiVersion: apps/v1 kind: Deployment spec: replicas: 4 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 # At most 1 extra pod during rollout maxUnavailable: 0 # Never reduce below desired count template: spec: containers: - name: api readinessProbe: httpGet: path: /health/ready port: 8000 initialDelaySeconds: 5 periodSeconds: 10 livenessProbe: httpGet: path: /health/live port: 8000 initialDelaySeconds: 15 periodSeconds: 20 terminationGracePeriodSeconds: 30 ``` **When to use:** Standard deploys of stateless services with backward-compatible changes. Most deploys. **Risk:** Old and new code run simultaneously. Database schema and API contracts must be backward-compatible. ### Blue-Green (Zero-Downtime with Instant Rollback) Run two identical environments. Deploy to the inactive one ("green"), verify, then switch traffic. Rollback is instant — switch traffic back to "blue." ```bash # Cloud Run example — deploy to a new revision without serving traffic gcloud run deploy my-service \ --image gcr.io/my-project/my-service:${NEW_SHA} \ --no-traffic \ --tag canary # Smoke test the new revision curl -s https://canary---my-service-xxxxx.a.run.app/health/ready # If healthy, shift all traffic gcloud run services update-traffic my-service --to-latest # If broken, rollback to previous revision gcloud run services update-traffic my-service \ --to-revisions=my-service-00042-abc=100 ``` ```yaml # Docker Compose blue-green (simplified) # nginx.conf switches upstream between blue and green services: blue: image: myapp:current ports: ["8001:8000"] green: image: myapp:${NEW_TAG} ports: ["8002:8000"] nginx: volumes: - ./nginx.conf:/etc/nginx/nginx.conf ports: ["80:80"] ``` **When to use:** Critical services where rollback speed matters more than resource cost. Services with hard availability SLAs. **Cost:** Requires 2x infrastructure during deploy window. Both environments must be fully functional. ### Canary (High-Risk Changes) Route a small percentage of traffic to the new version. Monitor error rates and latency. Gradually increase traffic if metrics are healthy. ```yaml # Kubernetes canary with Istio VirtualService apiVersion: networking.istio.io/v1 kind: VirtualService spec: hosts: ["my-service"] http: - route: - destination: host: my-service subset: stable weight: 95 - destination: host: my-service subset: canary weight: 5 ``` ```bash # Cloud Run canary — send 5% of traffic to new revision gcloud run services update-traffic my-service \ --to-revisions=my-service-00043-def=5,my-service-00042-abc=95 # Monitor for 15 minutes, then promote or rollback # Check error rate: # - If canary error rate > 2x stable error rate, rollback immediately # - If canary latency p99 > 1.5x stable p99, rollback immediately ``` **When to use:** Large schema changes, major refactors, new integrations, anything where "it worked in staging" is not sufficient confidence. **Canary promotion criteria:** 1. Error rate within 1.5x of stable baseline for 15 minutes 2. Latency p99 within 1.5x of stable baseline 3. No new error types in logs 4. No increase in pod restarts ### Feature Flags (Gradual Rollout at Application Level) Decouple deploy from release. Ship the code, then enable the feature gradually. This is the safest approach for user-facing changes. ```python # Python — LaunchDarkly / Unleash / simple feature flag import structlog logger = structlog.get_logger() def get_recommendations(user_id: str, feature_flags: FeatureFlags) -> list: if feature_flags.is_enabled("new_recommendation_engine", user_id=user_id):