jaw-dev-devops

name	jaw-dev-devops
description	MUST USE for infrastructure and delivery work — container builds, deploy pipelines, Kubernetes, Infrastructure as Code, SRE foundations, edge/serverless, ML infrastructure. Triggers: Dockerfile, K8s manifests, CI/CD pipeline, Terraform/IaC, release/deploy, devops/infra/deploy or release_cd task_tags.
metadata	{"short-description":"Container builds, deploy pipelines, K8s, IaC, SRE, edge/serverless, ML infra.","keywords":"container, kubernetes, deploy, iac, sre, pipeline, supply chain, sbom, gateway api","last-verified":"2026-07-02"}

Dev-DevOps — Production Infrastructure & Delivery

Build reliable, secure, and automated infrastructure and delivery pipelines. This skill has modular references for specialized guidance — read the relevant ones before coding.

C0/C1 work (small local patches): See dev §0.0 Work Classifier + §0.1 Patch Fast-Path before reading references.

dev is canonical: dev §0.2 Rule Classes, §3 Verification Gate, and §5 Safety Rules apply to all work governed by this skill.

Severity mapping: CRITICAL/HIGH ⇒ STRICT; MEDIUM ⇒ DEFAULT (aligned with dev §0.2).

Modular References

File	When to Read	What It Covers
references/docker.md	Container build/deploy	Multi-stage builds, distroless, Docker Scout/Trivy, BuildKit secrets, SBOM/Cosign
references/package-release.md	Package publishing / release auth	npm/PyPI trusted publishing, Bun-to-npm, registry auth model, downstream distribution table
references/cross-platform-release.md	Cross-platform release proof	CI matrix vs local OS proof, Windows App/RDP prompts, desktop verification boundaries
references/homebrew.md	Homebrew distribution	Formula vs Cask, audit/test, livecheck, artifact trust, install/uninstall proof
references/platform-engineering.md	Platform / DORA / provider routing	DORA capabilities, platform guardrails, provider table rows, SLSA handoff
references/kubernetes.md	K8s deployment	Gateway API (v1.6+), Kustomize overlays, HPA/VPA, Helm, ArgoCD GitOps
references/ci-cd-deploy.md	Deploy pipeline	GHA reusable workflows, deploy strategies, rollback, GitOps, progressive delivery
references/iac.md	Infrastructure code	OpenTofu/Terraform modules, Pulumi, state encryption, blast radius isolation
references/sre-foundations.md	Operations/incidents	SLO/SLI/error budget, burn-rate alerting, incident response, blameless postmortem
references/edge-serverless.md	Edge/serverless work	Edge request shaping, auth at edge, Cloudflare Workers, Vercel Edge, edge AI triage
references/ml-infra.md	ML infrastructure	GPU cluster mgmt, model registry, scaling, edge inference, MLOps platform patterns

Read package-release.md for package publishing, registry auth, npm/PyPI trusted publishing, Bun-to-npm release decisions, and downstream package channels. Read cross-platform-release.md when a release claim depends on OS-local behavior that CI may not prove. Read homebrew.md for Formula/Cask distribution work. Read platform-engineering.md for broader DevOps capability refresh, DORA, provider routing, and platform guardrails. Read docker.md + ci-cd-deploy.md first for containerized deploy workflows. For K8s-specific work, add kubernetes.md. For SRE/on-call, add sre-foundations.md.

When release, registry-auth, provider-doc, service-status, image/platform version, or package-manager behavior depends on current external evidence, read the active search skill and follow its source-fetch and evidence-status rules instead of relying on stale memory or copied snippets.

---

§1 Container Builds

§1.1 Dockerfile Rules (STRICT)

Rule	Detail
Multi-stage	Separate build and runtime stages; final image has no build tools
Base image	Pin version + SHA256 digest: node:22-slim@sha256:abc...
Minimal runtime base	Prefer gcr.io/distroless/* (Debian-based, keyless-signed) — no shell/package manager; OR Chainguard/Wolfi images when nightly rebuilds, SBOM attestations, patch SLAs, or compliance matter
Non-root	USER nonroot:nonroot (distroless) or create dedicated user
Dependency-first copy	COPY package.json bun.lock ./ → install → COPY . . for layer caching
BuildKit secrets	RUN --mount=type=secret,id=token ... — never use ARG for secrets
.dockerignore	.git, node_modules, .env*, *.log, dist/, coverage/, __pycache__/

For canonical Dockerfile templates, read references/docker.md §1.

§1.2 Image Security & Supply-Chain Baseline (STRICT)

CRITICAL/HIGH findings → block push. No exceptions. The 2026 baseline is ONE workflow, not separate tips: minimal base image (§1.1) → SBOM generation (Syft / Docker Scout) → vulnerability scan (Trivy or Grype) → sign + attest (Cosign/Sigstore) → digest-based promote (§2.4). Read references/docker.md §4 for scan/SBOM/sign command examples, and ../jaw-dev-security/references/supply-chain-sbom.md for deeper SBOM/signing policy.

§1.3 Anti-Patterns

Banned	Symptom	Fix
FROM node:latest	Irreproducible builds	Pinned version + digest
USER root in final stage	Attack surface	Non-root user
COPY . . as first instruction	Cache invalidation on every change	Dependency files first
ARG SECRET=xxx	Exposed in image history	BuildKit --mount=type=secret
No scan before push	Vulnerable images in prod	Trivy/Scout CI gate
apt-get install without cleanup	Bloated image	--no-install-recommends && rm -rf /var/lib/apt/lists/*

---

§2 Deploy Pipeline

§2.1 Pipeline Stages (DEFAULT)

[dev-testing §5]  lint → typecheck → test → contract → e2e
[dev-devops]      build-image → scan → push-registry → deploy-staging → smoke → promote → deploy-prod

§2.2 GHA Reusable Workflows (DEFAULT)

# .github/workflows/ci.yml (caller)
jobs:
  build:
    uses: org/templates/.github/workflows/build-test.yml@v2
    with:
      service: payments
    secrets: inherit

Rule	Detail
workflow_call	Central CI template, max 10-level nesting
Permissions	Caller cannot escalate; downgrade only
Environment	environment: production + required reviewers + prevent self-review
Promote	Digest-based (image@sha256:...), never mutable tags

§2.3 Deploy Strategies (DEFAULT)

Strategy	Tool	When	Risk
Rolling update	K8s Deployment	Stateless, low risk	Low
Blue-green	Argo Rollouts blueGreen:	Instant rollback, no DB migration	Medium
Canary	Argo Rollouts canary: steps:	Traffic % control, metric-based promote	Medium
Progressive	Flagger	Auto analysis + rollback, A/B testing	Medium-High
Feature flag	LaunchDarkly / Unleash	Code-level gradual rollout	Low

§2.4 Rollback Rules (STRICT)

• Every deployment must be rollback-capable within 5 minutes
• Digest-based promote only — mutable tags are banned
• DB migrations: forward-only + backward-compatible (expand-contract pattern)
• Post-rollback: automatic Slack/PagerDuty notification

§2.5 Secret Management (STRICT)

Source	Usage
GHA Secrets / Vault / AWS SM	CI pipeline secrets
External Secrets Operator	K8s → Vault/AWS SM sync
.env files	Never committed — generated in CI
Rotation	90-day cycle or immediate on incident

§2.6 GitOps (DEFAULT)

• Actions = CI, ArgoCD = CD — separation of concerns
• Actions updates deploy repo (image digest PR/commit) → ArgoCD reconciles
• Self-heal: ArgoCD auto-reverts drift
• Environment protection: GitHub Environments for prod approval gate

---

§3 Kubernetes Basics

§3.1 Minimum Viable K8s (DEFAULT)

Resource	Purpose
Deployment	Pod template + replica management
Service	Internal networking
HTTPRoute (Gateway API)	External traffic routing — not Ingress
ResourceQuota	Request/limit enforcement
Probes	Liveness + readiness + startup
Namespace	Environment isolation (dev/staging/prod)

§3.2 Gateway API (v1.6+, verified 2026-07-02)

Gateway API is the successor for new routing while Ingress remains GA but feature-frozen (not planned for removal). v1.6.0 (2026-06) graduates TCPRoute and UDPRoute to GA. Role separation: platform team owns GatewayClass + Gateway, app team owns HTTPRoute. Progressive delivery pairs directly with it: Flagger supports Gateway API HTTPRoute canaries.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: payments-route
spec:
  parentRefs:
    - name: shared-gateway
  hostnames: ["payments.example.com"]
  rules:
    - matches:
        - path: { type: PathPrefix, value: /api }
      backendRefs:
        - name: payments-svc
          port: 8080

§3.3 Scaling (DEFAULT)

Mechanism	Scope	Watch
HPA	CPU/memory + custom metrics	Don't combine with VPA on same metric
VPA	Request auto-tuning	Use Off mode for recommendations only
PDB	Disruption budget	minAvailable: 50% or maxUnavailable: 1

§3.4 Anti-Patterns

Banned	Fix
Ingress (new projects)	Gateway API HTTPRoute
No resource limits	Always set requests + limits
image: app:latest	SHA digest or pinned SemVer
Single replica in prod	Minimum 2 + PDB
Secrets in ConfigMap	K8s Secret + External Secrets Operator
Annotation-based routing	Gateway API native fields

---

§4 Infrastructure as Code

§4.1 OpenTofu/Terraform Rules (DEFAULT)

Rule	Detail
State	Remote backend required (S3+DynamoDB / TF Cloud / OpenTofu)
Encryption	OpenTofu native state/plan encryption via KMS
Blast radius	Separate state per app/layer/env
Modules	Purpose-built (vpc, iam, ecs-service), typed I/O
Apply	plan → PR review → apply; auto-apply staging only
Versions	Pin provider + module versions explicitly

§4.2 Tool Selection (HEURISTIC)

Tool	Best For	Status (verified 2026-07-02)
OpenTofu (HCL)	Open-source/licensing-neutral IaC default (MPL-2.0, Linux Foundation; v1.12.x)	✅ Recommended for OSS neutrality
Terraform / HCP Terraform (BSL)	Vendor support or HashiCorp platform integration required	✅ Active (BSL license — check competitive-use terms)
Pulumi (TS/Python)	Teams preferring programming languages	✅ Active
AWS CDK	AWS-only infrastructure	✅ Active (AWS only)
CDKTF	—	❌ Deprecated 2025-12-10; repo archived/read-only, no further fixes

§4.3 Anti-Patterns

Banned	Fix
Local state file	Remote backend required
Manual console changes	All changes via code
Hardcoded values	Variables + tfvars
Monolithic main.tf	Modular decomposition
CDKTF (new projects)	OpenTofu or Pulumi
Unpinned provider versions	Explicit version constraints

---

§5 SRE Foundations

§5.1 SLO/SLI (DEFAULT)

SLI	Measurement	Typical SLO
Availability	Success requests / total	99.9% (28-day rolling)
Latency	p50/p95/p99 response time	p99 < 500ms
Error rate	5xx / total	< 0.1%
Freshness	Data update delay	< 5min (pipelines)

SLIs measure user experience, not infrastructure metrics. "CPU is fine ≠ users are fine."

Error budget = 1 − SLO (99.9% → 0.1% budget).

DORA 2025 (verified 2026-07-02): AI acts as an amplifier — returns depend on the underlying sociotechnical system. For AI-agent-heavy delivery, invest in golden paths, guardrails, observability, provenance, and review gates, not just faster merges.

§5.2 Error Budget Policy (DEFAULT)

Budget State	Action
Normal (>50%)	Continue releases, routine monitoring
Accelerated burn (20–50%)	Heightened alerts, slow releases, reliability triage
Exhausted (≤0%)	Feature freeze — security/bugfix only; VP exception required

Single incident consuming >20% → mandatory postmortem. Two consecutive window misses → architecture review.

§5.3 Incident Response (DEFAULT)

1. Detect → Triage (S1/S2/S3) → Stabilize → Fix → Postmortem
2. Roles: IC, Primary Responder, Comms Lead, Scribe
3. Mitigation first, diagnosis second during active incidents
4. All S1/S2 → mandatory blameless postmortem within 5 business days
5. Status updates: S1 every 15min, S2 every 30min

§5.4 Runbook Template (HEURISTIC)

## [Service] — [Symptom]
### Diagnosis
1. Logs: `kubectl logs -l app=<name> --tail=100`
2. Metrics: Grafana → [dashboard URL]
3. Dependencies: `curl -s http://<dep>/health | jq .`
### Emergency Mitigation
1. Rollback: `argocd app rollback <app>`
2. Traffic block: ...
### Root Fix
1. ...
### Escalation
- Owner: @team-sre
- PagerDuty: [policy]

§5.5 Anti-Patterns

Banned	Fix
Infrastructure-only SLIs	User-experience-based SLIs
SLO without consequences	Error budget policy with freeze gate
Too many SLIs (>5 per service)	2-4 meaningful SLIs
Page on every deviation	Burn-rate multi-window alerting
Blame individuals	Blameless postmortem, system improvement
No error budget policy	Define 3-stage policy (normal/accelerated/exhausted)

---

§6 Cross-References

Topic	Canonical Owner	What dev-devops defers
Test strategy & CI test stages	dev-testing §5	Test pyramid, coverage gates
Backend observability code patterns	dev-backend observability.md	OTel SDK setup, structured logging
Security hardening (app-layer)	dev-security	OWASP, auth, input validation
SBOM/signing depth	dev-security references/supply-chain-sbom.md	Supply-chain evidence policy beyond image scan gates
Architecture module boundaries	dev-architecture	Coupling taxonomy, barrel discipline
Scaffolding conventions	dev-scaffolding	File naming, project structure
Frontend build/bundle	dev-frontend	Vite/webpack config, SSR

dev-devops owns: container builds, deploy pipelines, K8s manifests, IaC modules, SRE/incident response, edge infra, ML infra. DevOps owns operational scan execution and release gates; dev-security owns security policy, severity thresholds, and required evidence. dev-backend owns: application-layer observability code, API design, health check implementation. Overlap: observability alerting rules (dev-devops §5) ↔ observability code instrumentation (dev-backend observability.md). Cross-ref both.

Pre-flight Checklist

Before submitting infrastructure changes:

• Dockerfile is multi-stage with distroless/slim final image and non-root user
• Image scanned (Trivy/Scout) with CRITICAL/HIGH gate — no unresolved findings
• SBOM generated and attestation signed (Cosign) for production images
• Deploy pipeline uses digest-based promotion, never mutable tags
• K8s manifests have resource requests/limits, probes, PDB, and use Gateway API (not Ingress)
• IaC uses remote state, pinned versions, and modular decomposition
• Secrets are managed through Vault/AWS SM/GHA Secrets — no .env commits, no ARG secrets
• SLO/SLI defined with error budget policy and burn-rate alerting
• Rollback plan documented and tested — <5min rollback capability confirmed
• Runbook exists for critical failure scenarios