jaw-dev-devops

# Dev-DevOps — Production Infrastructure & Delivery Build reliable, secure, and automated infrastructure and delivery pipelines. This skill has modular references for specialized guidance — read the relevant ones before coding. > **C0/C1 work (small local patches):** See `jaw-dev` §0.0 Work Classifier + §0.1 Patch Fast-Path before reading references. > **`jaw-dev` is canonical:** `jaw-dev` §0.2 Rule Classes, §3 Verification Gate, and §5 Safety Rules apply to all work governed by this skill. Severity and rule authority are distinct (`jaw-dev` §0.2). Safety, correctness, and release proof remain mandatory; architecture and tool preferences need project-specific justification. ## Modular References | File | When to Read | What It Covers | |------|-------------|----------------| | `references/docker.md` | Container build/deploy | Multi-stage builds, distroless, Docker Scout/Trivy, BuildKit secrets, SBOM/Cosign | | `references/package-release.md` | Package publishing / release auth | npm/PyPI trusted publishing, Bun-to-npm, registry auth model, downstream distribution table | | `references/cross-platform-release.md` | Cross-platform release proof | CI matrix vs local OS proof, Windows App/RDP prompts, desktop verification boundaries | | `references/native-desktop-acceptance.md` | Electron desktop changes or release | UI, packaged runtime, artifact identity, signing and distribution evidence | | `references/macos-system-approvals.md` | macOS system prompts | Human approval boundary, read-only checks, evidence for blocked rows | | `references/repo-bootstrap.md` | Repository settings | Read-back audit, rulesets, merge choices and PR limits | | `references/agent-pr-intake.md` | Agent-contributed PRs | Identity, draft policy and supersede proof | | `references/local-gc.md` | Local worktree cleanup | PR-state evidence, active/dirty guards and dry-run schedule | | `../jaw-dev/references/stacked-prs.md` | Stacked PRs | Canonical layer and CI rules; opt in to native stack tooling only where supported | | `references/homebrew.md` | Homebrew distribution | Formula vs Cask, audit/test, livecheck, artifact trust, install/uninstall proof | | `references/platform-engineering.md` | Platform / DORA / provider routing | DORA capabilities, platform guardrails, provider table rows, SLSA handoff | | `references/kubernetes.md` | K8s deployment | Gateway API (v1.6+), Kustomize overlays, HPA/VPA, Helm, ArgoCD GitOps | | `references/ci-cd-deploy.md` | Deploy pipeline, and freeze/GO evidence | GHA reusable workflows, deploy strategies, rollback, GitOps, progressive delivery; §6 suite partitioning, baseline-vs-defect, instrument stability, exact-head evidence, stability counting | | `references/branch-lifecycle.md` | Branch or worktree cleanup | Deletion-evidence checks, ref snapshot and restore, merged-vs-closed gap, stacked-PR bases, fork identity by repository id, worktree removal order | | `references/iac.md` | Infrastructure code | OpenTofu/Terraform modules, Pulumi, state encryption, blast radius isolation | | `references/sre-foundations.md` | Operations/incidents | SLO/SLI/error budget, burn-rate alerting, incident response, blameless postmortem | | `references/edge-serverless.md` | Edge/serverless work | Edge request shaping, auth at edge, Cloudflare Workers, Vercel Edge, edge AI triage | | `references/ml-infra.md` | ML infrastructure | GPU cluster mgmt, model registry, scaling, edge inference, MLOps platform patterns | Read `package-release.md` for package publishing, registry auth, npm/PyPI trusted publishing, Bun-to-npm release decisions, and downstream package channels. Read `cross-platform-release.md` when a release claim depends on OS-local behavior that CI may not prove. Read `homebrew.md` for Formula/Cask distribution work. Read `platform-engineering.md` for broader DevOps capability refresh, DORA, provider routing, and platform guardrails. Read `docker.md` + `ci-cd-deploy.md` first for containerized deploy workflows. For K8s-specific work, add `kubernetes.md`. For SRE/on-call, add `sre-foundations.md`. When release, registry-auth, provider-doc, service-status, image/platform version, or package-manager behavior depends on current external evidence, read the active `search` skill and follow its source-fetch and evidence-status rules instead of relying on stale memory or copied snippets. --- ## §1 Container Builds ### §1.1 Dockerfile Rules (STRICT) | Rule | Detail | |------|--------| | Multi-stage | Separate build and runtime stages; final image has no build tools | | Base image | Pin version + SHA256 digest: `node:22-slim@sha256:abc...` | | Minimal runtime base | Prefer `gcr.io/distroless/*` (Debian-based, keyless-signed) — no shell/package manager; OR Chainguard/Wolfi images when nightly rebuilds, SBOM attestations, patch SLAs, or compliance matter | | Non-root | `USER nonroot:nonroot` (distroless) or create dedicated user | | Dependency-first copy | `COPY package.json bun.lock ./` → install → `COPY . .` for layer caching | | BuildKit secrets | `RUN --mount=type=secret,id=token ...` — never use `ARG` for secrets | | `.dockerignore` | `.git`, `node_modules`, `.env*`, `*.log`, `dist/`, `coverage/`, `__pycache__/` | For canonical Dockerfile templates, read `references/docker.md` §1. ### §1.2 Image Security & Supply-Chain Baseline (STRICT) CRITICAL/HIGH image findings block push. A later checklist exception cannot waive this gate; any policy change must be approved and declared before the verdict. Other checklist exceptions follow their owning policy. The 2026 baseline is ONE workflow, not separate tips: minimal base image (§1.1) → SBOM generation (Syft / Docker Scout) → vulnerability scan (Trivy or Grype) → sign + attest (Cosign/Sigstore) → digest-based promote (§2.4). Read `references/docker.md` §4 for scan/SBOM/sign command examples, and `../jaw-dev-security/references/supply-chain-sbom.md` for deeper SBOM/signing policy. ### §1.3 Anti-Patterns | Banned | Symptom | Fix | |--------|---------|-----| | `FROM node:latest` | Irreproducible builds | Pinned version + digest | | `USER root` in final stage | Attack surface | Non-root user | | `COPY . .` as first instruction | Cache invalidation on every change | Dependency files first | | `ARG SECRET=xxx` | Exposed in image history | BuildKit `--mount=type=secret` | | No scan before push | Vulnerable images in prod | Trivy/Scout CI gate | | `apt-get install` without cleanup | Bloated image | `--no-install-recommends && rm -rf /var/lib/apt/lists/*` | --- ## §2 Deploy Pipeline ### §2.1 Pipeline Stages (DEFAULT) ``` [jaw-dev-testing §5] lint → typecheck → test → contract → e2e [jaw-dev-devops] build-image → scan → push-registry → deploy-staging → smoke → promote → deploy-prod ``` For stacked PRs, use `../jaw-dev/references/stacked-prs.md` for layer shape (`DEV-STACK-03`), membership (`DEV-STACK-06`), CI diagnosis (`DEV-STACK-07`), and the native-tool opt-in (`DEV-STACK-OPT-IN-01`). ### §2.2 GHA Reusable Workflows (DEFAULT) ```yaml # .github/workflows/ci.yml (caller) jobs: build: uses: org/templates/.github/workflows/build-test.yml@v2 with: service: payments secrets: inherit ``` | Rule | Detail | |------|--------| | `workflow_call` | Central CI template, max 10-level nesting | | Permissions | Caller cannot escalate; downgrade only | | Environment | `environment: production` + required reviewers + prevent self-review | | Promote | Digest-based (`image@sha256:...`), never mutable tags | ### §2.3 Deploy Strategies (DEFAULT) | Strategy | Tool | When | Risk | |----------|------|------|------| | Rolling update | K8s Deployment | Stateless, low risk | Low | | Blue-green | Argo Rollouts `blueGreen:` | Instant rollback, no DB migration | Medium | | Canary | Argo Rollouts `canary: steps:` | Traffic % control, metric-based promote | Medium | | Progressive | Flagger | Auto analysis + rollback, A/B testing | Medium-High | | Feature flag | LaunchDarkly / Unleash | Code-level gradual rollout | Low | ### §2.4 Rollback Rules (STRICT) - Every deployment must be rollback-capable within 5 minutes - Digest-based promote only — mutable tags are banned - DB migrations: forward-only + backward-compatible (expand-contract pattern) - Post-rollback: automatic Slack/PagerDuty notification ### §2.5 Secret Management (STRICT) **Rule (DEVOPS-AUTH-01):** prefer OIDC, workload identity federation, trusted publishing, or another short-lived credential flow **before** static long-lived tokens. When a static token is unavoidable, scope it narrowly, store it in the managed secret system, and rotate on schedule or on incident. | Source | Usage | |--------|-------| | GHA Secrets / Vault / AWS SM | CI pipeline secrets | | External Secrets Operator | K8s → Vault/AWS SM sync | | `.env` files | **Never committed** — generated in CI | | Rotation | 90-day cycle or immediate on incident | ### §2.6 GitOps (DEFAULT) - **Actions = CI, ArgoCD = CD** — separation of concerns - Actions updates deploy repo (image digest PR/commit) → ArgoCD reconciles - Self-heal: ArgoCD auto-reverts drift - Environment protection: GitHub Environments for prod approval gate ### §2.7 Release Proof Contract (STRICT) **Rule (DEVOPS-RELEASE-PROOF-01):** a release claim must name the artifact digest, the workflow or builder identity, the deploy target and environment, smoke-test evidence, and rollback evidence. Five items, each an artifact rather than an assertion — "deployed successfully" names none of them. Keep the proof at this level; detailed package, platform, and supply-chain mechanics live in the release references. ### §2.8 Freeze and GO/NO-GO Gates (STRICT) `DEVOPS-RELEASE-PROOF-01` governs the proof bundle for an artifact you already published. This section governs the decision to publish at all — the readiness report, and the gates it claims to have passed. | Rule | Severity | Statement | |------|----------|-----------| | `DEVOPS-FREEZE-SHA-01` | STRICT | Pin the readiness report to the code SHA its gates describe. If the head moved after the freeze, prove the delta is docs-only (`git diff --name-only <freeze> <head>`) and keep every gate receipt on the freeze SHA. | | `DEVOPS-GATE-WEAKEN-01` | STRICT | A red named gate is never excused inside the report that gate failed. Make the original command green, or replace it with a pre-declared equivalent CI actually runs — and declare the swap **before** the verdict, not after the failure. | | `DEVOPS-REVIEW-THREADS-01` | STRICT | Unresolved review threads on merged PRs are a GO blocker. Count them **after** merge: a thread opened minutes before merge still counts until it is fixed or explicitly dismissed. | | `DEVOPS-GATE-OWNER-01` | STRICT | A mandatory GO gate needs an implementing work-phase and a recorded terminal outcome — pass, not-reproduced, or explicitly deregistered. A gate nobody implements is not a gate; it is a wish. | **Why gate-weakening is the load-bearing one.** In the case that produced these rules, a freeze audit rejected the first GO report on three counts: unresolved review threads on merged PRs, a red gate argued into an exception, and missing frozen-head receipts. The red-gate part is the instructive one — the suite was red, the red tests were known to be load-sensitive, and the fix was real, so the report explained the exception. The audit rejected that anyway, and correctly: **an exception argued *after* a gate fails is indistinguishable from an exception argued *because* it failed.** The honest move was to decompose the gate to match what CI actually runs (`DEVOPS-SUITE-PARTITION-01`), which turned the general suite green. The one remaining local failure was waived separately on the `DEVOPS-BASELINE-DEFECT-01` triple — not by the partitioning. Two different mechanisms for two different problems; collapsing them is how a partition becomes a way to hide a defect. Operational mechanics — suite partitioning, baseline-versus-defect attribution, instrument stability, and exact-head evidence — live in `references/ci-cd-deploy.md` §6. Runtime and operator-signal evidence rules live in `references/sre-foundations.md` §7. ### §2.9 Branch Lifecycle Hygiene (STRICT) This guidance does not itself authorize repository-setting changes, scheduled jobs, branch or worktree deletion. Apply the task's actual write authority. Delivery repositories accumulate dead refs, and the cost is not disk. Stale branches make the remote branch list unusable for triage, keep superseded heads reachable by tooling that resolves names, and hide the handful of branches that actually still matter. Treat branch lifecycle as delivery infrastructure. | Rule | Severity | Statement | |------|----------|-----------| | `DEVOPS-BRANCH-AUTODELETE-01` | STRICT | Enable host-side head deletion on merge (`delete_branch_on_merge` on GitHub) **and** close the gap it leaves. That setting fires only on merge; a pull request closed without merging keeps its head branch forever, so the closed-PR case needs its own scheduled automation. | | `DEVOPS-BRANCH-DELETE-EVIDENCE-01` | STRICT | Never bulk-prune. Before deleting any ref, prove per branch that it is not protected, not an open PR head, not the base of an open PR, not a fork head, and not carrying unique commits. A name pattern is not evidence. | | `DEVOPS-BRANCH-SNAPSHOT-01` | STRICT | Snapshot `git for-each-ref` (SHA + refname) for every local and remote ref to scratch space before the first deletion. A deleted remote branch is restorable with `git push origin <sha>:refs/heads/<name>` only while you still hold the SHA. | | `DEVOPS-WORKTREE-DIRTY-01` | STRICT | Check every attached worktree for uncommitted work before removing it, and remove worktrees **before** their branches — an attached branch cannot be deleted, and `--force` on a dirty tree discards work no reflog will return. | | `DEVOPS-BRANCH-NAMESPACE-01` | STRICT | Closed-PR cleanup needs a repository-declared disposable prefix and an exact match between current tip and a related closed PR head SHA; see `branch-lifecycle.md`. | | `DEVOPS-REPO-BOOTSTRAP-01` | DEFAULT | Read back repository settings before proposing changes; see `repo-bootstrap.md`. | | `DEVOPS-AGENT-INTAKE-01` | DEFAULT | Use the repository's declared PR review policy; see `agent-pr-intake.md`. | | `DEVOPS-LOCAL-GC-01` | STRICT | Classify by PR state and SHA; snapshot refs, exclude active/dirty/locked or uncertain trees, and dry-run before cleanup; see `local-gc.md`. | **Why the merged/closed distinction is load-bearing.** `delete_branch_on_merge` reads as complete branch hygiene, so a repository with it enabled looks solved. It is not: in the case behind this rule the setting was on and the repository still carried 59 dead remote branches, because closed-unmerged PRs fall outside what it covers. The failure is silent and compounds — nothing reports it, and the branch list simply degrades until triage stops using it. **Why stacked PRs break naive cleanup.** A stacked child PR targets its parent's head branch, so deleting a closed parent **closes the open child**. "The PR that owned this branch is closed" is therefore insufficient grounds for deletion: the base of any open PR is protected regardless of its own PR state (`jaw-dev` `DEV-STACK-04`). **Fork heads are out of scope, and identity is by repository id.** A fork's head lives in the contributor's repository; deleting refs there is neither permitted nor intended. Compare repository **ids**, not names — a fork commonly carries the same branch names as upstream, so name comparison silently misclassifies it. Mechanics, the deletion-plan algorithm, and a worked audit live in `references/branch-lifecycle.md`. Repository policy choices, contributed PR review and local worktree cleanup live in `references/repo-bootstrap.md`, `references/agent-pr-intake.md` and `references/local-gc.md`. --- ## §3 Kubernetes Basics ### §3.1 Minimum Viable K8s (DEFAULT) | Resource | Purpose | |----------|---------| | Deployment | Pod template + replica management | | Service | Internal networking | | HTTPRoute (Gateway API) | External traffic routing — **not Ingress** | | ResourceQuota | Request/limit enforcement | | Probes | Liveness + readiness + startup | | Namespace | Environment isolation (dev/staging/prod) | ### §3.2 Gateway API (v1.6+, verified 2026-07-02) Gateway API is the successor for new routing while Ingress remains GA but feature-frozen (not planned for removal). v1.6.0 (2026-06) graduates TCPRoute and UDPRoute to GA. Role separation: platform team owns `GatewayClass` + `Gateway`, app team owns `HTTPRoute`.