devops-deployment

# DevOps & Deployment Skill Comprehensive frameworks for CI/CD pipelines, containerization, deployment strategies, and infrastructure automation. ## When to Use - Setting up CI/CD pipelines - Containerizing applications - Deploying to Kubernetes or cloud platforms - Implementing GitOps workflows - Managing infrastructure as code - Planning release strategies ## Pipeline Architecture ``` ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Code │──▶│ Build │──▶│ Test │──▶│ Deploy │ │ Commit │ │ & Lint │ │ & Scan │ │ & Release │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │ │ ▼ ▼ ▼ ▼ Triggers Artifacts Reports Monitoring ``` ## Key Concepts ### CI/CD Pipeline Stages 1. **Lint & Type Check** - Code quality gates 2. **Unit Tests** - Test coverage with reporting 3. **Security Scan** - npm audit + Trivy vulnerability scanner 4. **Build & Push** - Docker image to container registry 5. **Deploy Staging** - Environment-gated deployment 6. **Deploy Production** - Manual approval or automated > See `templates/github-actions-pipeline.yml` for complete GitHub Actions workflow ### Container Best Practices **Multi-stage builds** minimize image size: - Stage 1: Install production dependencies only - Stage 2: Build application with dev dependencies - Stage 3: Production runtime with minimal footprint **Security hardening**: - Non-root user (uid 1001) - Read-only filesystem where possible - Health checks for orchestrator integration > See `templates/Dockerfile` and `templates/docker-compose.yml` ### Kubernetes Deployment **Essential manifests**: - Deployment with rolling update strategy - Service for internal routing - Ingress for external access with TLS - HorizontalPodAutoscaler for scaling **Security context**: - `runAsNonRoot: true` - `allowPrivilegeEscalation: false` - `readOnlyRootFilesystem: true` - Drop all capabilities **Resource management**: - Always set requests and limits - Use `requests` for scheduling, `limits` for throttling > See `templates/k8s-manifests.yaml` and `templates/helm-values.yaml` ### Deployment Strategies | Strategy | Use Case | Risk | |----------|----------|------| | **Rolling** | Default, gradual replacement | Low - automatic rollback | | **Blue-Green** | Instant switch, easy rollback | Medium - double resources | | **Canary** | Progressive traffic shift | Low - gradual exposure | **Rolling Update** (Kubernetes default): ```yaml strategy: type: RollingUpdate rollingUpdate: maxSurge: 25% maxUnavailable: 0 # Zero downtime ``` **Blue-Green**: Deploy to standby environment, switch service selector **Canary**: Use Istio VirtualService for traffic splitting (10% → 50% → 100%) ### Infrastructure as Code **Terraform patterns**: - Remote state in S3 with DynamoDB locking - Module-based architecture (VPC, EKS, RDS) - Environment-specific tfvars files > See `templates/terraform-aws.tf` for AWS VPC + EKS + RDS example ### GitOps with ArgoCD ArgoCD watches Git repository and syncs cluster state: - Automated sync with pruning - Self-healing (drift detection) - Retry policies for transient failures > See `templates/argocd-application.yaml` ### Secrets Management Use External Secrets Operator to sync from cloud providers: - AWS Secrets Manager - HashiCorp Vault - Azure Key Vault - GCP Secret Manager > See `templates/external-secrets.yaml` ## Deployment Checklist ### Pre-Deployment - [ ] All tests passing in CI - [ ] Security scans clean - [ ] Database migrations ready - [ ] Rollback plan documented ### During Deployment - [ ] Monitor deployment progress - [ ] Watch error rates - [ ] Verify health checks passing ### Post-Deployment - [ ] Verify metrics normal - [ ] Check logs for errors - [ ] Update status page ## Helm Chart Structure ``` charts/app/ ├── Chart.yaml ├── values.yaml ├── templates/ │ ├── deployment.yaml │ ├── service.yaml │ ├── ingress.yaml │ ├── configmap.yaml │ ├── secret.yaml │ ├── hpa.yaml │ └── _helpers.tpl └── values/ ├── staging.yaml └── production.yaml ``` --- ## CI/CD Pipeline Patterns ### Branch Strategy **Recommended: Git Flow with Feature Branches** ``` main (production) ─────●────────●──────▶ ┃ ┃ dev (staging) ─────●───●────●───●──────▶ ┃ ┃ feature/* ─────────●────────┘ ▲ └─ PR required, CI checks, code review ``` **Branch protection rules:** - `main`: Require PR + 2 approvals + all checks pass - `dev`: Require PR + 1 approval + all checks pass - Feature branches: No direct commits to main/dev ### GitHub Actions Caching Strategy ```yaml - name: Cache Dependencies uses: actions/cache@v3 with: path: | ~/.npm node_modules backend/.venv key: ${{ runner.os }}-deps-${{ hashFiles('**/package-lock.json', '**/poetry.lock') }} restore-keys: | ${{ runner.os }}-deps- ``` **Cache hit ratio impact:** - Without cache: 2-3 min install time - With cache: 10-20 sec install time - **~85% time savings** on typical workflows ### Artifact Management ```yaml # Build and upload artifact - name: Build Application run: npm run build - name: Upload Build Artifact uses: actions/upload-artifact@v3 with: name: build-${{ github.sha }} path: dist/ retention-days: 7 # Download in deployment job - name: Download Build Artifact uses: actions/download-artifact@v3 with: name: build-${{ github.sha }} path: dist/ ``` **Benefits:** - Avoid rebuilding in deployment job - Deploy exact tested artifact (byte-for-byte match) - Retention policies prevent storage bloat ### Matrix Testing ```yaml strategy: matrix: node-version: [18, 20, 22] os: [ubuntu-latest, windows-latest] jobs: test: runs-on: ${{ matrix.os }} steps: - uses: actions/setup-node@v3 with: node-version: ${{ matrix.node-version }} - run: npm test ``` --- ## Container Optimization Deep Dive ### Multi-Stage Build Example ```dockerfile # ============================================================ # Stage 1: Dependencies (builder) # ============================================================ FROM node:20-alpine AS deps WORKDIR /app COPY package*.json ./ RUN npm ci --only=production && npm cache clean --force # ============================================================ # Stage 2: Build (with dev dependencies) # ============================================================ FROM node:20-alpine AS builder WORKDIR /app COPY package*.json ./ RUN npm ci # Include dev dependencies COPY . . RUN npm run build && npm run test # ============================================================ # Stage 3: Production runtime (minimal) # ============================================================ FROM node:20-alpine AS runner WORKDIR /app # Security: Non-root user RUN addgroup -g 1001 -S nodejs && adduser -S nodejs -u 1001 # Copy only production dependencies and built artifacts COPY --from=deps --chown=nodejs:nodejs /app/node_modules ./node_modules COPY --from=builder --chown=nodejs:nodejs /app/dist ./dist COPY --chown=nodejs:nodejs package*.json ./ USER nodejs EXPOSE 3000 ENV NODE_ENV=production HEALTHCHECK --interval=30s --timeout=3s CMD node healthcheck.js || exit 1 CMD ["node", "dist/main.js"] ``` **Image size comparison:** - Single-stage: **850 MB** (includes dev dependencies, source files) - Multi-stage: **180 MB** (only runtime + production deps) - **78% reduction** ### Layer Caching Optimization **Order matters for cache efficiency:** ```dockerfile # ❌ BAD: Invalidates cache on any code change COPY . . RUN npm install # ✅ GOOD: Cache package.json layer separately COPY package*.json ./ RUN npm ci # Cached unless package.json changes COPY . . # Source changes don't invalidate npm install ``` ### Security Scanning with Trivy ```yaml - name: Build Docker Image run: docker build -t myapp:${{ github.sha }} . - name: Scan for Vulnerabilities uses: aquasecurity/trivy-action@master with: image-ref: 'myapp:${{ github.sha }}' format: 'sarif' output: 'trivy-results.sarif' severity: 'CRITICAL,HIGH' - name: Upload Scan Results uses: github/codeql-action/upload-sarif@v2 with: sarif_file: 'trivy-results.sarif' - name: Fail on Critical Vulnerabilities run: | trivy image --severity CRITICAL --exit-code 1 myapp:${{ github.sha }} ``` --- ## Kubernetes Production Patterns ### Health Probes **Three probe types with distinct purposes:** ```yaml spec: containers: - name: app # Startup probe (gives slow-starting apps time to boot) startupProbe: httpGet: path: /health/startup port: 8080 initialDelaySeconds: 0 periodSeconds: 5 failureThreshold: 30 # 30 * 5s = 150s max startup time # Liveness probe (restarts pod if failing) livenessProbe: httpGet: path: /health/liveness port: 8080 initialDelaySeconds: 60 periodSeconds: 10 failureThreshold: 3 # 3 failures = restart # Readiness probe (removes from service if failing) readinessProbe: httpGet: path: /health/readiness port: 8080 initialDelaySeconds: 10 periodSeconds: 5 failureThreshold: 2 # 2 failures = remove from load balancer ``` **Probe implementation:** ```python @app.get("/health/startup") async def startup_check(): # Check DB connection established if not db.is_connected(): raise HTTPException(status_code=503, detail="DB not ready") return {"status": "ok"} @app.get("/health/liveness") async def liveness_check(): # Basic "is process running" check return {"status": "alive"} @app.get("/health/readiness") async def readiness_check(): # Check all dependencies healthy if not redis.ping() or not db.health_check(): raise HTTPException(status_code=503, detail="Dependencies unhealthy") return {"status": "ready"} ``` ### PodDisruptionBudget Prevents too many pods from being evicted during node maintenance: ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: app-pdb spec: minAvailable: 2 # Always keep at least 2 pods running selector: matchLabels: app: myapp ``` **Use cases:** - Cluster upgrades (node drains) - Autoscaler downscaling - Manual evictions ### Resource Quotas ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: team-quota namespace: production spec: hard: requests.cpu: "10" # Total CPU requests requests.memory: 20Gi # Total memory requests limits.cpu: "20" # Total CPU limits limits.memory: 40Gi # Total memory limits pods: "50" # Max pods ``` ### StatefulSets for Databases ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: postgres spec: serviceName: postgres replicas: 3 selector: matchLabels: app: postgres template: # Pod spec here volumeClaimTemplates: - metadata: name: data spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 100Gi ```