k8s-cluster-upgrade-runbook

name	k8s-cluster-upgrade-runbook
enabled	true
description	Use when performing k8s cluster upgrade runbook — kubernetes cluster upgrade runbook covering pre-upgrade checks, API deprecation review, node upgrade procedure, validation tests, and rollback steps. Supports EKS, GKE, AKS, and self-managed clusters. Use for minor/major version upgrades.
required_connections	[{"prefix":"kubernetes","label":"Kubernetes Cluster"}]
config_fields	[{"key":"cluster_name","label":"Cluster Name","required":true,"placeholder":"e.g., prod-us-east-1"},{"key":"current_version","label":"Current Version","required":true,"placeholder":"e.g., 1.28"},{"key":"target_version","label":"Target Version","required":true,"placeholder":"e.g., 1.29"},{"key":"distribution","label":"Distribution","required":false,"placeholder":"e.g., EKS, GKE, AKS, kubeadm"}]
features	["KUBERNETES","DEPLOYMENT"]

Kubernetes Cluster Upgrade Runbook Skill

Upgrade cluster {{ cluster_name }} from {{ current_version }} to {{ target_version }} ({{ distribution | "self-managed" }}).

Workflow

Step 1 — Pre-Upgrade Assessment

PRE-UPGRADE CHECKS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
CLUSTER HEALTH
[ ] All nodes in Ready state
[ ] No pods in CrashLoopBackOff or Pending state
[ ] Cluster autoscaler healthy (if enabled)
[ ] etcd cluster healthy (self-managed only)
[ ] Control plane components healthy
[ ] PodDisruptionBudgets reviewed (won't block node drain)

API COMPATIBILITY
[ ] Reviewed deprecated APIs between {{ current_version }} and {{ target_version }}
[ ] No workloads using removed APIs
[ ] Ran: kubectl deprecations (or pluto scan)
[ ] Helm charts compatible with {{ target_version }}
[ ] CRDs compatible with target version
[ ] Admission webhooks compatible

ADD-ONS & COMPONENTS
[ ] CoreDNS version compatible
[ ] kube-proxy version compatible
[ ] CNI plugin compatible (Calico, Cilium, VPC-CNI)
[ ] CSI drivers compatible
[ ] Ingress controller compatible
[ ] Cert-manager compatible
[ ] Monitoring stack compatible (Prometheus, etc.)

Step 2 — Backup & Snapshot

BACKUP
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] etcd snapshot taken (self-managed clusters)
[ ] Cluster state exported (kubectl get all --all-namespaces -o yaml)
[ ] PV snapshots taken for critical volumes
[ ] Helm release manifests backed up
[ ] Custom resource definitions backed up
[ ] Backup verified and stored securely

Step 3 — Control Plane Upgrade

CONTROL PLANE UPGRADE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
For managed (EKS/GKE/AKS):
[ ] Initiate control plane upgrade via cloud provider
[ ] Monitor upgrade progress
[ ] Verify API server responding with target version

For self-managed:
[ ] Upgrade kubeadm on first control plane node
[ ] Run: kubeadm upgrade plan
[ ] Run: kubeadm upgrade apply {{ target_version }}
[ ] Upgrade kubelet and kubectl on control plane nodes
[ ] Restart kubelet on each control plane node
[ ] Verify all control plane nodes on target version

Step 4 — Worker Node Upgrade

WORKER NODE UPGRADE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Strategy: [ ] Rolling (one at a time)  [ ] Blue-green (new node group)

For each node (or node group):
[ ] Cordon node: kubectl cordon <node>
[ ] Drain node: kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
[ ] Verify pods rescheduled to other nodes
[ ] Upgrade node (AMI swap / kubeadm upgrade)
[ ] Uncordon node: kubectl uncordon <node>
[ ] Verify node Ready with target version
[ ] Verify pods running on upgraded node
[ ] Wait for stability (5 min) before next node

Progress tracking:
  Nodes upgraded: ___/___
  Nodes remaining: ___

Step 5 — Add-On Upgrades

ADD-ON UPGRADES
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] CoreDNS upgraded to compatible version
[ ] kube-proxy upgraded to {{ target_version }}
[ ] CNI plugin upgraded (if needed)
[ ] CSI drivers upgraded (if needed)
[ ] Ingress controller upgraded (if needed)
[ ] Cert-manager upgraded (if needed)
[ ] Monitoring agents upgraded (if needed)

Step 6 — Post-Upgrade Validation

VALIDATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
CLUSTER HEALTH
[ ] All nodes Ready and on target version
[ ] All system pods healthy (kube-system namespace)
[ ] DNS resolution working (test with nslookup from pod)
[ ] Service discovery working
[ ] Cluster autoscaler functioning

WORKLOAD HEALTH
[ ] All deployments at desired replica count
[ ] No pods in error state
[ ] Application health checks passing
[ ] Ingress/load balancer routing correctly
[ ] Persistent volumes mounted and accessible

FUNCTIONALITY
[ ] Pod scheduling working (create test pod)
[ ] Horizontal Pod Autoscaler responding
[ ] Network policies enforced
[ ] RBAC working correctly
[ ] Logging and monitoring data flowing

Step 7 — Rollback Procedure

ROLLBACK (if critical issues found)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Managed clusters:
[ ] Initiate control plane downgrade (if supported)
[ ] Roll back node groups to previous AMI/version

Self-managed:
[ ] Restore etcd from snapshot
[ ] Downgrade kubeadm, kubelet, kubectl
[ ] Verify cluster state restored

All:
[ ] Verify workloads healthy after rollback
[ ] Document reason for rollback
[ ] Plan remediation before retry

Counter-Rationalizations

Shortcut	Counter	Why
"We can skip some steps for this case"	Adapt the workflow steps, don't skip them	Skipped steps are where incidents and oversights originate
"The user seems to already know what to do"	Complete all workflow phases with the user	The workflow catches blind spots that experience alone misses
"This is a minor case, full process is overkill"	Scale the process down, don't turn it off	Minor cases become major when unstructured; the process scales, not disappears
"I'll fill in the details later"	Complete each section before moving on	Deferred details are forgotten; real-time capture is more accurate
"The template output isn't necessary"	Always produce the structured output format	Structured output enables comparison, audit trails, and handoff to other teams

Output Format

Produce an upgrade execution report with:

1. Pre-upgrade assessment with compatibility check results
2. Upgrade execution log with timestamps per step
3. Node upgrade progress tracker
4. Validation results with PASS/FAIL per check
5. Issues encountered with resolution or rollback decision