kubernetes

# Kubernetes ## Pod security - Never run as root. `runAsNonRoot: true`, `runAsUser` (non-zero) in `securityContext` - `readOnlyRootFilesystem: true`. Mount writable volumes explicitly - `allowPrivilegeEscalation: false` - Drop all capabilities, re-add only what's needed: `capabilities: {drop: [ALL], add: [NET_BIND_SERVICE]}` - No `hostPID`, `hostNetwork`, `hostIPC` without explicit system-level justification - Pod Security Admission: `restricted` profile for all application workloads. `baseline` minimum for system workloads - `automountServiceAccountToken: false` on pods that don't call the API server ## RBAC - Least privilege. Separate `ServiceAccount` per workload. No default SA with cluster-level roles - `Role`/`RoleBinding` over `ClusterRole`/`ClusterRoleBinding` unless cluster-scope is required - Audit RBAC with `kubectl auth can-i --list` and `rakkess` or `rbac-tool` - No `verbs: ["*"]` or `resources: ["*"]` in production roles ## Secrets - Secrets in external store (Vault, AWS Secrets Manager, GCP Secret Manager). Sync via External Secrets Operator or Secrets Store CSI driver - Never commit secrets to git. Plain Kubernetes `Secret` objects are base64 only — require etcd encryption at rest - etcd encryption at rest enabled. etcd backed up on schedule (Velero or equivalent) ## Network policy - Default-deny ingress and egress per namespace. Explicit allow rules per service - NetworkPolicies enforced by CNI (Calico, Cilium, or equivalent). Verify CNI supports policy enforcement — not all do - Cilium: use `CiliumNetworkPolicy` for L7 rules (HTTP, gRPC path/method filtering) where needed ## Workloads - `Deployment` for stateless. `StatefulSet` for stateful with stable identity. `DaemonSet` for node-level agents - `requests` and `limits` on every container. No unbounded CPU or memory - CPU `limits` cause throttling — set conservatively or omit and rely on namespace `LimitRange`. Memory `limits` cause OOMKill — set with headroom - `minReplicas` ≥ 2 for production. Single-replica = not HA - Liveness probe: deadlock detection. Readiness probe: traffic gating. Startup probe: slow-starting containers. All three on every long-running container - `terminationGracePeriodSeconds` set to cover max request duration + drain time. Default 30s is often too short - `preStop: exec: sleep 5` (or equivalent) to allow load balancer to drain before SIGTERM - `PodDisruptionBudget` on all production workloads. `minAvailable` > 0 - `topologySpreadConstraints` or `podAntiAffinity` to spread replicas across nodes and zones - `HorizontalPodAutoscaler` on CPU/memory or custom metrics for stateless workloads - KEDA for event-driven scaling (queue depth, Kafka lag, cron). Preferred over custom metrics pipelines - `VerticalPodAutoscaler` in recommendation mode to right-size requests over time ## Images - Never `latest` tag in manifests. Pin to digest or immutable tag - `imagePullPolicy: Always` for mutable tags. `imagePullPolicy: IfNotPresent` for digest-pinned images - Don't build images in-cluster. Build → scan → push → deploy (see Docker skill) ## Configuration - `ConfigMap` for non-sensitive config. External secrets (not raw K8s `Secret`) for sensitive data - No env-specific values baked into images. All config injected at runtime - Namespaces per environment and team. `ResourceQuota` and `LimitRange` per namespace - Labels on all resources: `app.kubernetes.io/name`, `app.kubernetes.io/version`, `app.kubernetes.io/component`, `env`, `team` ## Networking and ingress - Ingress controller (NGINX, Traefik) for HTTP/HTTPS. Gateway API preferred for new clusters — more expressive than `Ingress` - TLS termination at ingress. `cert-manager` for certificate lifecycle (Let's Encrypt or internal CA) - Internal service-to-service: `ClusterIP`. External: `LoadBalancer` or ingress only. Never `NodePort` in production ## Policy enforcement - OPA/Gatekeeper or Kyverno for org-wide policy as code: enforce labels, block `latest`, require probes, restrict privileges - Policies in version control. Applied via GitOps, not ad-hoc - Admission webhooks audited — failing-open webhooks can silently bypass policy ## Operations - GitOps (ArgoCD or Flux) as single source of truth. No direct `kubectl apply` to production - `kubectl diff` before any manual apply. Never apply blindly - Helm or Kustomize for templating. No raw generated manifests in CI - Cluster Autoscaler or Karpenter for node scaling. Right-size node pools per workload class (spot, on-demand, GPU) - API server audit logs enabled. Ship to centralised logging - Falco for runtime threat detection (unexpected syscalls, shell in container, sensitive file access) - `kubectl rollout` for deploys. `maxUnavailable: 0`, `maxSurge: 1` for zero-downtime - Upgrade clusters within N-2 of latest minor. Never run unsupported versions