eks-upgrade-readiness

# EKS Upgrade Readiness Assess and plan Amazon EKS cluster upgrades with comprehensive pre-upgrade validation aligned with the [EKS Best Practices Guide](https://docs.aws.amazon.com/eks/latest/best-practices/cluster-upgrades.html). ## When to Use Activate this skill when the user asks to: - Check if an EKS cluster is ready to upgrade - Plan an EKS version upgrade (control plane, node groups, or both) - Identify deprecated Kubernetes APIs before upgrading - Validate addon compatibility with a target version - Assess node group upgrade strategy and capacity requirements - Review Pod Disruption Budgets or topology spread for upgrade safety - Understand EKS end-of-support, extended support, or auto-upgrade implications - Evaluate Karpenter Drift or node expiry upgrade behavior - Compare in-place vs blue-green upgrade strategies - Create an upgrade runbook or checklist - Detect GitOps/IaC version ownership before upgrading ## Safety First **Before doing anything, load `references/safety-invariants.md`.** It defines the knowledge hierarchy, hard rules, operation classification, and uncertainty handling. Keep it in context for the entire assessment. ## Critical Warnings - **This skill is read-only.** All commands are `describe*`, `list*`, `get*`. The agent does NOT execute mutating APIs. Mutations are in Step 14 and require explicit operator approval. - **One minor version at a time.** EKS control plane upgrades proceed one minor version per operation (e.g., 1.30 → 1.31). - **Version skew policy.** Before planning an upgrade, no kubelet may be newer than the **current** control plane. For the target version, kubelet may be no more than N-3 on 1.28+ (N-2 below 1.28). - **Addons must be upgraded AFTER the control plane** (exceptions in Step 8). - **Auto-upgrade policy.** Clusters past the 26-month lifecycle will be auto-upgraded. Proactive upgrade avoids disruption. - **Control plane rollback (July 2026+).** 7-day rollback window after upgrade. Conditional, not guaranteed — skill checks eligibility. - **UNKNOWN ≠ PASS.** Any gate that cannot be assessed MUST be UNKNOWN, never PASS. Overall verdict cannot be READY while any gate is UNKNOWN. ## Evidence Completeness Uses `references/required-check-registry.yaml` to track checks performed, skipped, or blocked. EC = checks_performed / total_applicable × 100%. EC < 50% produces a mandatory warning. ## Grading and Confidence | Level | Meaning | When to Use | |-------|---------|-------------| | HIGH (90%+) | Confirmed from authoritative source | EKS Insights API, direct kubectl query, AWS API response | | MEDIUM (60-89%) | Inferred from available data | Partial kubectl access, version matching heuristics | | LOW (30-59%) | Limited data, possible gaps | No kubectl, no logging enabled, partial API access | | UNKNOWN | Cannot determine | Tool unavailable, no data, access denied | **False-positive guards:** - Empty query result ≠ PASS (mark UNKNOWN) - No kubectl ≠ N/A for everything (AWS APIs still work) - EKS Insights PASSING ≠ skip other checks (covers a subset only) - Addon "compatible" ≠ "recommended" - Pagination not exhausted → confidence LOW **Verdict rules (evaluate applicable gates only; `N/A` gates are excluded):** 1. **NOT READY**: one or more applicable gates are FAIL. A known blocker wins over uncertainty because proceeding is unsafe. 2. **CANNOT DETERMINE**: no gate is FAIL, but one or more applicable gates are UNKNOWN (including inaccessible, incomplete, stale, or unpaginated data). 3. **READY WITH WARNINGS**: all applicable gates are assessed, none FAIL or UNKNOWN, and one or more are WARN. 4. **READY**: every applicable gate is PASS. Format: `[PASS|FAIL|WARN|UNKNOWN|N/A] (confidence: HIGH) — <evidence>` ## Cost Awareness - **EKS Insights API** (Step 3) is free — always use first. - **CloudWatch Logs Insights** cost ~$0.0076/GB scanned. Default to 60-min windows. - **Extended support** costs $0.60/cluster/hour — upgrading saves money. - **Surge nodes** incur temporary EC2 cost during overlap period. ## Required Permissions **AWS IAM** — see README.md "Prerequisites → IAM Permissions" for the full read-only action list (`eks:Describe*`, `eks:List*`, `ec2:Describe*`, `autoscaling:Describe*`, `iam:GetRole`, `servicequotas:GetServiceQuota`). **Kubernetes RBAC** (only if `kubectl` access is available — the assessment still runs on AWS APIs alone without it, at lower confidence for CRD/Helm/PDB checks). Read-only `ClusterRole` covering every `kubectl get`/`describe` used in this skill: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: eks-upgrade-readiness-readonly rules: - apiGroups: [""] resources: - nodes - pods - configmaps - secrets - events - persistentvolumeclaims - certificatesigningrequests verbs: ["get", "list", "watch"] - apiGroups: ["apps"] resources: ["deployments", "statefulsets", "daemonsets", "replicasets"] verbs: ["get", "list", "watch"] - apiGroups: ["policy"] resources: ["poddisruptionbudgets"] verbs: ["get", "list", "watch"] - apiGroups: ["apiextensions.k8s.io"] resources: ["customresourcedefinitions"] verbs: ["get", "list", "watch"] - apiGroups: ["admissionregistration.k8s.io"] resources: - validatingwebhookconfigurations - mutatingwebhookconfigurations verbs: ["get", "list", "watch"] - apiGroups: ["karpenter.sh"] resources: ["nodepools", "nodeclaims"] verbs: ["get", "list", "watch"] - apiGroups: ["karpenter.k8s.aws"] resources: ["ec2nodeclasses"] verbs: ["get", "list", "watch"] - apiGroups: ["crd.k8s.amazonaws.com"] resources: ["eniconfigs"] verbs: ["get", "list", "watch"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses", "csinodes"] verbs: ["get", "list", "watch"] ``` Bind with a `ClusterRoleBinding` to the identity the agent assumes (e.g. via IRSA/Pod Identity or an EKS access entry). `secrets` read access is required only for the Helm stored-manifest scan (Step 4) — omit that rule and accept UNKNOWN on Helm checks if a customer's security policy disallows it. --- ## Step 1: Gather Cluster Context ```bash aws eks describe-cluster --name <cluster-name> --region <region> ``` Extract: `cluster.version`, `platformVersion`, `status` (must be ACTIVE), `kubernetesNetworkConfig`, `logging.clusterLogging` (audit log must be enabled), `resourcesVpcConfig.subnetIds`, `tags` (IaC ownership detection). Determine **target version**: ask user or default to current + 1 minor. Confirm target is in standard support via the EKS release calendar. ## Step 2: Verify Infrastructure Prerequisites Check these — failures are **BLOCKERs**: 1. **Subnet IP availability** — need ≥5 IPs per cluster subnet. Mode-aware: standard IPv4, prefix delegation, custom networking, IPv6, SGP. Use `aws ec2 describe-subnets` with cluster subnet IDs. 2. **EKS IAM role** — verify role exists with `eks.amazonaws.com` trust. 3. **KMS key** (if encryption enabled) — verify cluster role has key access. 4. **Service quota headroom** — EC2 vCPU (L-1216C47A) and EBS gp3 (L-7A658000) must have room for surge nodes. Use `aws service-quotas get-service-quota`. VPC CNI mode and capacity-input detection: ```bash kubectl get ds aws-node -n kube-system -o json | jq ' .spec.template.spec.containers[0].env[] | select(.name | test("ENABLE_PREFIX_DELEGATION|AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG|ENABLE_POD_ENI|WARM_IP_TARGET|MINIMUM_IP_TARGET|WARM_ENI_TARGET|WARM_PREFIX_TARGET")) | {name, value}' ``` ### VPC CNI Surge-Capacity Gate Do not treat "mode detected" as capacity validated. First calculate the managed-node-group surge using `references/capacity-planning.md`, distribute it by the node group's AZ placement, then verify the relevant subnet/ENI resource for the selected mode. Record the inputs and calculations as evidence; a missing mode-specific input is `UNKNOWN`, not PASS. | Mode | Required assessment before a node surge | Pass condition | |------|------------------------------------------|----------------| | Standard IPv4 | Inspect `WARM_IP_TARGET`, `MINIMUM_IP_TARGET`, and `WARM_ENI_TARGET` on `aws-node`; use node `status.allocatable.pods` and current pod count to calculate the additional secondary-IP demand for every surge node. | Every node subnet has enough free IPv4 addresses for its share of surge nodes, their primary ENIs, and configured warm/allocatable pod-IP demand. | | Prefix delegation | Confirm `ENABLE_PREFIX_DELEGATION=true`; each IPv4 prefix consumes a `/28` (16 addresses). Calculate required additional prefixes as `ceil(additional_pod_ips / 16)` per affected subnet/AZ. | `floor(availableIpAddressCount / 16)` covers the needed prefixes after allowing for node primary addresses and the configured warm-prefix target. | | Custom networking | Confirm `AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true`, enumerate `ENIConfig` resources, and map each node AZ to its `spec.subnet` and security groups. | The **ENIConfig pod subnet**, not only the cluster/node subnet, has capacity for the surge pod-IP demand in every used AZ. | | Security Groups for Pods | Confirm `ENABLE_POD_ENI=true`, inspect trunk ENIs and the instance-type-specific branch-ENI limit for each node type. | Required branch ENIs/pod slots for surge workloads with pod SGs do not exceed the published limit for any instance type. Do not use generic ENI limits as a substitute. | | IPv6 | Confirm IPv6 family and Nitro-compatible node/Fargate support. IPv6 pod addressing does not consume IPv4 pod IPs, but nodes still need valid ENI/subnet capacity. | Node ENI and subnet capacity cover surge nodes; custom networking is not assumed because it is unsupported with IPv6. | ```bash # Custom networking: inspect every AZ-to-pod-subnet mapping kubectl get eniconfig -o json | jq '.items[] | {name: .metadata.name, subnet: .spec.subnet, securityGroups: .spec.securityGroups}' # Security Groups for Pods: inspect trunk/branch interfaces after mode detection aws ec2 describe-network-interfaces \ --filters Name=interface-type,Values=trunk,branch \ --query 'NetworkInterfaces[].{type:InterfaceType,subnet:SubnetId,instance:Attachment.InstanceId,status:Status}' ``` ### Client and CI Tooling Skew (WARN-level, not a blocker) Operator/CI tooling that is too far behind the target version causes confusing failures during and after the upgrade. Check installed versions where available: ```bash kubectl version --client -o json # client minor version eksctl version # if eksctl-managed helm version --short # if Helm-managed workloads ``` | Tool | Skew Rule | Risk if Violated | |------|-----------|-------------------| | kubectl | Must be within ±1 minor of the target `kube-apiserver` version (upstream [Kubernetes version skew policy](https://kubernetes.io/releases/version-skew-policy/#kubectl)) | Unrecognized fields, API calls silently rejected or misinterpreted | | eksctl | Must support the target EKS version (check release notes for the version that added support) | `eksctl upgrade` commands fail or use stale defaults | | Helm | 3.8+ recommended for OCI registry support; otherwise not EKS-version-gated | Chart operations may fail independent of the cluster upgrade | | Terraform AWS provider | Must be new enough to support any target-version-specific attributes in use (e.g. `upgrade_policy`, `compute_config` for Auto Mode) — check the [provider changelog](https://github.com/hashicorp/terraform-provider-aws/blob/main/CHANGELOG.md) for the attribute | `terraform apply` fails validation or silently ignores the new attribute | Do not hardcode exact version floors here — they shift every EKS release. Report the installed version, the rule, and a WARN if it cannot be confirmed current; never treat "tool not detected" as PASS. ## Step 3: Check EKS Upgrade Insights **Primary authoritative signal.** Always query first. ```bash aws eks list-insights --cluster-name <cluster> \ --filter '{"categories":["UPGRADE_READINESS"],"kubernetesVersions":["<target>"]}' aws eks describe-insight --cluster-name <cluster> --id <insight-id> ``` | Status | Gate | Action | |--------|------|--------| | ERROR | FAIL | Must fix before upgrade | | WARNING | WARN | Recommended fix | | PASSING | PASS | No action | | UNKNOWN | UNKNOWN | EKS could not evaluate the check; investigate and refresh | | None returned | UNKNOWN | Continue other checks | **Freshness gate:** For every returned summary, capture `lastRefreshTime` and `lastTransitionTime`, then call `DescribeInsight` to collect the status, affected resources, and recommendation. Treat data as **stale** when `lastRefreshTime` is more than 24 hours old at assessment time, or predates a known relevant workload/addon change. A stale, missing, inaccessible, or unpaginated insight set is `UNKNOWN`; never reuse it as PASS. The assessment must not call `StartInsightsRefresh` because this skill is read-only. Instead, ask the operator to refresh insights through an approved workflow and rerun the assessment after the refresh completes. **Critical:** Insights does NOT cover Helm stored manifests, CRD deprecations, StatefulSets, Karpenter, service quotas, PDBs, or capacity planning. ## Step 4: API Deprecation Analysis Check version-specific removal gates relevant to user's target: - ≥1.33: AL2 AMI unavailable (Critical) - ≥1.35: kube-proxy IPVS deprecated; ≥1.36: removed - ≥1.25: Dockershim and PodSecurityPolicy removed - ≥1.23: In-tree EBS provisioner deprecated **Helm stored manifests** — the #1 missed blocker. Scan latest deployed release secrets for deprecated `apiVersion` lines: ```bash kubectl get secrets -A -l owner=helm,status=deployed # Decode: base64 -d | base64 -d | gunzip | jq -r '.manifest' ``` **Third-party CRD deprecations** — check Istio, cert-manager, Karpenter, Flux, Argo, Prometheus Operator versions against known deprecation timelines. Tools: `kubent`, `pluto detect-all-in-cluster`, `helm mapkubeapis --dry-run`. See `references/api-deprecations.md` for full removal schedule. ## Step 5: Addon Compatibility Check **Managed addons:** Use live API to build compatibility matrix: ```bash aws eks list-addons --cluster-name <cluster> aws eks describe-addon --cluster-name <cluster> --addon-name <name> aws eks describe-addon-versions --addon-name <name> --kubernetes-version <target> ``` **Self-managed addons:** First compare `ListAddons` with the actual `kube-system` workloads. Explicitly detect the three core addons: `aws-node` (VPC CNI), `coredns`, and `kube-proxy`. If a core component is absent from the managed-addon inventory but present in-cluster, mark it self-managed/custom and inspect its image, args, and configuration before assessing target support: ```bash kubectl -n kube-system get daemonset aws-node kube-proxy -o json | \ jq '.items[] | {name: .metadata.name, images: [.spec.template.spec.containers[].image], args: [.spec.template.spec.containers[].args]}' kubectl -n kube-system get deployment coredns -o json | \