Menu
Guidance on managing the lifecycle and upgrades of Google Kubernetes Engine (GKE) clusters.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Systematically diagnose GKE JobSet interruptions, restarts, and preemptions for AI/ML training workloads. Identifies preemption events, maintenance interruptions, bad host VMs, unhealthy pods, and coordinator worker failures.
Diagnose and predict node disruption during Compute Engine host maintenance for GPU and TPU workloads.
Diagnose and prevent `vbar_control_agent` segfaults and OOMs caused by race conditions during TPU device resets and frequent metrics collection (e.g. every 3s). Use when TPU slice initialization fails or `vbar_control_agent` crashes on TPU v6e nodes.
Verifies if a GKE or Kubernetes cluster is unused (no active compute, external exposure, or persistent data) before allowing deletion. Evaluates external exposure (LoadBalancer Service, Ingress, Gateway, MultiClusterIngress), persistent data (Bound PVC), and active compute (Running/Pending Pods in user namespaces) with low-overhead queries and fail-close timeouts.
Monitor and manage GKE TPU Dynamic Slices custom resources. Use when checking slice lifecycle states, troubleshooting failed slice creations (e.g. SliceCreationFailed, FAILED), running single or multi-slice workloads, or safely deleting/disabling slices.
Monitor and troubleshoot GKE TPU workloads using GKE system metrics and PromQL.
| name | gke-cluster-lifecycle |
| description | Guidance on managing the lifecycle and upgrades of Google Kubernetes Engine (GKE) clusters. |
This skill provides guidance on managing the lifecycle and upgrades of Google Kubernetes Engine (GKE) clusters.
Managing cluster upgrades is crucial for security and access to new features. GKE provides automated upgrades, but they must be configured to minimize disruption.
Release channels allow you to choose the balance between stability and feature availability.
Command to set release channel:
gcloud container clusters update <cluster-name> \
--release-channel=stable \
--region <region>
Surge upgrades allow you to specify how many nodes can be created above the target size during an upgrade, minimizing disruption.
Example configuration:
gcloud container node-pools update <pool-name> \
--cluster=<cluster-name> \
--max-surge-upgrade=2 \
--max-unavailable-upgrade=0 \
--region <region>
Setting max-unavailable-upgrade=0 ensures that no nodes are taken offline before new ones are ready.
For high-risk upgrades, you can create a new node pool (Green) with the new version, test it, and then migrate workloads from the old node pool (Blue).
Steps:
Stable or Regular).max-surge-upgrade to ensure availability during upgrades.