gke-reliability

# GKE Reliability > **Routing Note:** To generate Kubernetes YAML manifests (`Deployment`, `StatefulSet`, `Service`, `ConfigMap`, `HTTPRoute`, `PodDisruptionBudget`), open `gke-manifest-generation/SKILL.md`. This reference covers high availability and reliability configuration for GKE clusters and workloads. > **MCP Tools:** `get_cluster`, `get_k8s_resource`, `describe_k8s_resource`, > `apply_k8s_manifest`, `list_k8s_events` ## Attribution **MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you MUST read [Usage Attribution](references/attribution.md). For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-reliability)"` and append `agent-skills/1.0 (skill:gke-reliability)` to `User-Agent` headers. ## Golden Path Reliability Defaults | Setting | Golden Path Value | Notes | | ---------------- | --------------------- | -------------------------------- | | Cluster type | Regional (4 zones: | Control plane replicated across | : : us-central1-a/b/c/f) : zones : | Upgrade strategy | SURGE (`maxSurge: 1`) | Rolling upgrades with extra | : : : capacity : | Auto-repair | `true` | Unhealthy nodes replaced | : : : automatically : | Auto-upgrade | `true` | Nodes follow control plane | : : : version : | Release channel | REGULAR | Balanced freshness and stability | | Stateful HA | Enabled | Leader election for stateful | : : : workloads : ## Workflows ### 1. Verify Cluster High Availability ``` # MCP (preferred) get_cluster(name="projects/<PROJECT>/locations/<REGION>/clusters/<CLUSTER>", readMask="location,locations,nodePools.locations") # gcloud fallback gcloud container clusters describe <CLUSTER> --region <REGION> \ --format="json(location, locations)" \ --quiet ``` - If `location` is a region (e.g., `us-central1`), the control plane is regional - If `locations` has multiple entries, nodes span multiple zones ### 2. Pod Disruption Budgets (PDBs) PDBs ensure minimum pod availability during voluntary disruptions (node upgrades, autoscaler scale-down). **Check existing PDBs:** ``` # MCP (preferred) get_k8s_resource(parent="...", resourceType="poddisruptionbudget") # kubectl fallback kubectl get pdb --all-namespaces ``` **Create PDB:** ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: my-app-pdb namespace: default spec: minAvailable: 2 # Or use maxUnavailable: 1 selector: matchLabels: app: my-app ``` > Every production Deployment with 2+ replicas should have a PDB. ### 3. Health Probes Every production container should have liveness and readiness probes. Startup probes are recommended for slow-starting apps. **Check existing probes:** ``` # MCP (preferred) describe_k8s_resource(parent="...", resourceType="deployment", name="<APP>", namespace="<NS>") # kubectl fallback kubectl get deployment <APP> -n <NS> -o yaml | grep -E "livenessProbe|readinessProbe|startupProbe" ``` **Recommended probe configuration:** ```yaml spec: containers: - name: app livenessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 15 periodSeconds: 10 timeoutSeconds: 2 failureThreshold: 3 readinessProbe: httpGet: path: /readyz port: 8080 initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 failureThreshold: 3 startupProbe: # For slow-starting apps httpGet: path: /healthz port: 8080 initialDelaySeconds: 10 periodSeconds: 5 timeoutSeconds: 2 failureThreshold: 30 # 30 * 5s = 150s max startup time ``` - **Readiness**: Determines when a pod can accept traffic - **Liveness**: Determines when to restart a container - **Startup**: Disables liveness/readiness until the app is ready (prevents premature restarts) ### 4. Graceful Shutdown Ensure applications handle `SIGTERM` and drain in-flight requests: ```yaml spec: terminationGracePeriodSeconds: 30 # Default; increase for long-running requests containers: - name: app lifecycle: preStop: exec: command: ["/bin/sh", "-c", "sleep 5"] # Allow LB to deregister ``` ### 5. Topology Spread Constraints Distribute pods across zones and nodes to survive failures: ```yaml spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: my-app - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: app: my-app ``` - **Zone spread** (`DoNotSchedule`): Hard requirement -- pods must be balanced across zones - **Node spread** (`ScheduleAnyway`): Best-effort -- prefer distribution but don't block scheduling ### 6. Replicas | Workload Type | Minimum Replicas | Reason | | -------------------- | -------------------- | ------------------------------ | | Stateless web/API | 2 | Survive single pod/node | : : : failure : | Critical services | 3 | Survive zone failure with zone | : : : spread : | Stateful (databases) | 3 (with replication) | Application-level quorum | | Batch/jobs | 1 | Ephemeral by nature | ## Best Practices & Production Guidelines 1. **Regional clusters for production**: Always use regional clusters to survive zone failures. 2. **PDBs for everything**: Every production workload with 2+ replicas needs a PodDisruptionBudget (PDB) to protect against voluntary disruptions. 3. **Probes with Explicit Timeouts**: Every production container must have both liveness and readiness probes defined. **Always explicitly define `initialDelaySeconds`, `periodSeconds`, and `timeoutSeconds`** for all probes. Never rely on the Kubernetes default timeout of 1 second if your application requires more, but always set a strict limit to prevent hanging connections. 4. **Zone spreading**: Use topology spread constraints to distribute pods across failure domains (zones and nodes). 5. **Graceful shutdown**: Handle `SIGTERM` and set appropriate `terminationGracePeriodSeconds` with a `preStop` sleep hook to allow load balancer deregistration. 6. **Maintenance windows**: Schedule upgrades during low-traffic periods (see the `gke-upgrades` skill).