Create, review, and architect Kubernetes infrastructure - from raw manifests to Helm charts to multi-cluster strategy. The goal is production-ready, security-hardened, cost-aware infrastructure that a team can maintain.
Target versions (June 2026): Kubernetes 1.34-1.36 supported (1.36 "Haru" released April 22, 2026; 1.36.1 / 1.35.5 / 1.34.8 are the current patches). Upstream Kubernetes has no LTS; community support per minor is ~14 months. 1.33 entered maintenance April 28, 2026 and reaches upstream EOL June 28, 2026; 1.32 already hit upstream EOL February 28, 2026. Managed vendor extended support (AKS/EKS) carries older minors patches roughly 2 more years - attribute it to the platform, not upstream. Helm 4.2.1, Helm 3.21.x (parallel v3 maintenance, security fixes until Nov 2026).
This skill covers four domains depending on context:
Manifests - raw YAML for Deployments, Services, Gateway API routes, ConfigMaps, Secrets, PVCs
Scaffolding new Helm charts or improving existing ones
Designing cluster topology, GitOps strategy, or multi-tenancy
Implementing security contexts, network policies, RBAC, admission control
Setting up multi-environment deployments (dev/staging/prod)
Reviewing infrastructure for production or compliance readiness
Planning observability, cost optimization, or disaster recovery
PCI-DSS 4.0 compliance for fintech/payment K8s workloads
When NOT to use
Configuring CI/CD pipelines (use ci-cd)
Docker/container image optimization (use docker)
Security audits of application code (use security-audit)
Provisioning the cluster itself via IaC (use terraform)
Database engine configuration running on K8s (use databases)
Broad read-only cluster health checks, status reports, and post-maintenance diagnostics (use cluster-health)
AI Self-Check
This skill runs inside an AI agent. AI tools consistently produce the same K8s security mistakes. Before returning any generated manifest, verify against this list:
Security context present on every pod AND every container (not just one level)
Using Gateway API HTTPRoute for new external access, not legacy Ingress
Liveness and readiness probes defined: every container has at least a readiness probe
Kube context verified before any kubectl/helm/argocd command
Requester is authorized for cluster/admin changes, especially in shared chats. If the request comes from a non-admin participant, stop and ask the authorized owner for approval before kubectl, Helm, ArgoCD, or GitOps edits.
No auto-sync to production without approval gate
Current source checked: dated versions, CLI flags, API names, and support windows are verified against primary docs before repeating them
Hidden state identified: local config, credentials, caches, contexts, branches, cluster targets, or previous runs are made explicit before acting
Verification is real: final checks exercise the actual runtime, parser, service, or integration point instead of only linting prose or happy paths
Routing overlap checked: overlapping skills, trigger terms, and "When NOT to use" boundaries are checked before returning guidance
Spec claims verified: claims about tool behavior, output contracts, or repo conventions are checked against current docs, scripts, or skill files
API versions checked: manifests, Helm templates, and Gateway resources match the target cluster version
Cluster context verified: namespace, context, and kubeconfig identity are shown before mutating commands
kube-proxy mode checked on 1.35+ clusters: IPVS mode is deprecated in 1.35 (removal targeted for a future release); recommend nftables mode for new clusters and flag IPVS in reviews
Run generated manifests through kube-score, kubelinter, or checkov when available.
Performance
Set requests and limits from measured workload behavior; missing requests damage scheduling and autoscaling.
Use server-side dry-run and diff before apply; avoid repeated full-cluster renders during tight loops.
Scope watches, logs, and kubectl get calls by namespace/labels in large clusters.
Best Practices
Prefer declarative GitOps or reviewed manifests over live imperative changes for production.
Back up CRDs and custom resources before upgrades or operator changes.
Use policy gates for privileged pods, hostPath, broad RBAC, and mutable image tags.
Workflow
Step 1: Determine the domain
Based on the request:
"Create a deployment/service/manifest" -> Manifests
"Create a Helm chart" / "package for deployment" -> Helm
"Design the cluster" / "how should we structure" -> Architecture
"Make this PCI compliant" / "fintech" -> Compliance
"Review this manifest/chart" -> Apply production checklist + critical rules + AI self-check
Most real tasks blend domains. Work bottom-up: get the manifests right, then template them, then plan the deployment.
Step 2: Gather requirements
Before writing YAML, determine:
Workload type: stateless (Deployment) vs stateful (StatefulSet) vs batch (Job/CronJob)
Container image and pinned tag or SHA256 digest
Ports exposed (container port, service port, protocol)
Config: env vars, config files, secrets
Storage: ephemeral (emptyDir) vs persistent (PVC) with access mode and size
Resources: CPU/memory requests and limits
Health: startup, liveness, and readiness probe endpoints
Access: internal-only (ClusterIP) vs external (Gateway API HTTPRoute / LoadBalancer)
Scale: replicas, HPA thresholds, pod disruption budget
Step 5: GitOps-managed emergency or scaling changes
When changing a live workload managed by ArgoCD, Flux, or another reconciler, read references/gitops-emergency-changes.md; live kubectl scale, kubectl patch, or manual apply may be reverted unless the desired state changes too.
Manifests
Read references/manifest-templates.md for complete, copy-pasteable YAML templates (Deployment, Service, Gateway API HTTPRoute, ConfigMap, PVC, StatefulSet, native sidecar).
Key patterns
Labels: use the app.kubernetes.io/* standard labels on every resource:
app.kubernetes.io/name - app name
app.kubernetes.io/version - version string
app.kubernetes.io/component - role (frontend, backend, database)
app.kubernetes.io/part-of - parent system
External access (new clusters must use Gateway API, not legacy Ingress):
Gateway APIHTTPRoute (GA v1.5): role-oriented, expressive routing, no annotation hell. Ingress-NGINX retired March 2026 (its Feb 2026 RCE set CVE-2026-24512/24513/24514, CVSS 8.8, is a further reason to migrate off; fixed in ingress-nginx 1.13.7 / 1.14.3 if you must stay).
ClusterIP (default): internal-only
LoadBalancer: cloud LB without HTTP routing
Headless (clusterIP: None): StatefulSet pod discovery
Security context (non-negotiable on every pod - both pod-level AND container-level). See the Deployment template in manifest-templates.md for the full YAML. Key fields: runAsNonRoot, readOnlyRootFilesystem, allowPrivilegeEscalation: false, drop: ["ALL"], seccompProfile: RuntimeDefault.
Three probes (startup + liveness + readiness):
startupProbe: gates the other probes until the app is ready (high failureThreshold, moderate periodSeconds)
readinessProbe: removes from service endpoints (aggressive - pull traffic fast on failure)
Pod distribution: prefer topologySpreadConstraints over pod anti-affinity for zone-level distribution (anti-affinity is O(n^2) at scale). Combine both: topologySpreadConstraints for zones + soft anti-affinity for node-level separation within zones.
Native sidecars (GA in K8s 1.33): init containers with restartPolicy: Always. Start before main containers, stay running alongside them, terminate after main containers exit. Replaces all sidecar lifecycle hacks (preStop hooks, shareProcessNamespace kill scripts).
In-place pod resize (GA in K8s 1.35): CPU and memory can be updated on running pods without restart. VPA can now resize without disruption using InPlaceOrRecreate mode.
Config/secrets: ConfigMap for non-sensitive data. For secrets, use External Secrets Operator syncing from a vault/cloud KMS, or Sealed Secrets for encrypted-in-git workflows (see references/sealed-secrets.md). Never commit plaintext secrets anywhere.
Helm Charts
Helm 4 (4.0.0 released Nov 12, 2025; 4.2.1 current) is current. Helm 3.21.x gets security fixes until Nov 2026.
What changed in Helm 4
Server-side apply (SSA) is the default for new releases. Better conflict detection when multiple controllers touch the same resources.
WASM plugin system - post-renderers must reference plugin names, not raw executables (breaking change).
CLI flag renames: --atomic -> --rollback-on-failure, --force -> --force-replace. The old flags still work as deprecated aliases on both install and upgrade (a brief Helm 4.0 regression that broke helm install --atomic was fixed in v4.1.3); prefer --rollback-on-failure for new scripts.
helm registry login takes domain only (e.g., ghcr.io, not full URL).
OCI registries are the recommended distribution method. Traditional index.yaml repos still work but are no longer the default path.
Nested access: alias with {{- $var := .Values.deep.nested }} to avoid spaghetti
Helm anti-patterns
Hardcoded values in templates (should come from values.yaml)
tpl on static strings (it is for dynamic template rendering)
.Values.foo.bar.baz chains without default on optional values
Unpinned chart dependencies (use ~ ranges or exact versions)
No NOTES.txt
Missing .helmignore (test/ci files end up in package)
randAlphaNum in templates deployed via ArgoCD (causes perpetual OutOfSync)
Mega-umbrella charts (15+ subcharts, 200+ value overrides) - use ArgoCD ApplicationSets instead
replicaCount: 1 for production HA workloads (no redundancy, single point of failure)
authentication.enabled: false or persistence.enabled: false in production values (data loss, unauthorized access)
Secrets in Helm values (use ESO/sealed-secrets/vault; Helm values end up readable in cluster Secrets)
Hook resources without helm.sh/hook-delete-policy (orphaned Jobs accumulate)
Using --post-renderer with raw executables (broken in Helm 4; must use plugin names)
Ignoring SSA migration - upgrading to Helm 4 on existing releases can surface previously-hidden conflicts
ArgoCD + Helm caveats
ArgoCD only runs helm template - it does NOT use Helm lifecycle management. Don't rely on Helm hooks for critical operations; use ArgoCD sync waves instead.
test hooks are unsupported in ArgoCD.
Multi-source Applications (ArgoCD 2.6+) for separating chart version from environment values.
OCI charts: omit the oci:// prefix in ArgoCD's repoURL field.
Kustomize
Kustomize is built into kubectl (kubectl apply -k) and available as a standalone CLI.
Overlay pattern (base + environments)
app/
+-- base/
| +-- kustomization.yaml # references resources
| +-- deployment.yaml
| +-- service.yaml
+-- overlays/ # dev/ staging/ prod/, each a kustomization.yaml with bases: [../../base]
Each overlay's kustomization.yaml sets bases, then adds patches, images, and configMapGenerator/secretGenerator overrides for that environment.
Components
Reusable cross-cutting patches (monitoring, security context, sidecar injection) that any overlay can opt into with components: [../../components/monitoring]. Prefer components over duplicating patches across overlays.
secretGenerator / name-suffix-hash pitfall
secretGenerator and configMapGenerator append a content hash suffix to the resource name (e.g., app-config-abc12345). Deployments referencing the generated name by a fixed name break because the hash changes on every edit. Always reference generated resources by the base name and let Kustomize resolve the suffix, or set options.disableNameSuffixHash: true if the hash-based rolling update is not desired.
Architecture
Read references/architecture.md for the full architecture decision framework. Key patterns:
Cluster topology
Single cluster when: < 50 services, single team, single region, non-critical workloads.
Multi-cluster when: multi-region HA, team isolation, blast radius reduction, compliance boundaries (PCI CDE).
GitOps
ArgoCD when: UI matters, app-of-apps, multi-cluster from single control plane, RBAC on deployments.
Promotion: dev -> staging -> prod via PR-based promotion. No auto-sync to prod.
Networking
Gateway API (GA v1.5) is the standard for new clusters (see Rule 11).
CNI: Cilium (eBPF, greenfield) or Calico (brownfield/multi-OS/Windows). Cilium includes Hubble observability, L3-L7 policy, and optional sidecar-free service mesh.
kube-proxy: nftables mode is the future. IPVS deprecated in 1.35, removal targeted for a future release (no firm version committed yet).
Service mesh (add only when needed):
Istio ambient (GA in 1.24): sidecarless L4 mTLS via ztunnel, optional L7 via waypoint proxies. The "sidecars are too expensive" argument is dead.
Cilium: mTLS via WireGuard/IPsec, no mesh abstraction. Simpler but less L7 control.
Linkerd: stable builds now vendor-only (Buoyant). Source is Apache 2.0 but you build your own or pay.
Security (defense in depth)
8 layers for production:
Cluster hardening: CIS benchmark, API server audit logging, etcd encryption via KMS v2
Pod Security Standards: enforce: restricted is mandatory on all app namespaces - no exceptions. Set audit: restricted and warn: restricted everywhere else for visibility into what would break before enforcing.
Admission control: ValidatingAdmissionPolicy (CEL, native since 1.30) for standard policies; Kyverno for mutation/generation; OPA Gatekeeper for cross-platform orgs
Network policies: default-deny ingress/egress per namespace; Cilium for L7 policies
RBAC: namespace-scoped roles, no cluster-admin for apps, OIDC auth with MFA
Supply chain: cosign/Sigstore for image signing, SLSA Level 2-3, SBOMs. Pin all CI actions and tools to commit SHAs - the Trivy supply chain compromise (March 2026, CVE-2026-33634) proved mutable tags can be force-pushed with malware.
Secrets: External Secrets Operator + cloud KMS (primary); Vault for dynamic secrets/PKI; Sealed Secrets for encrypted-in-git without external deps (see references/sealed-secrets.md); SOPS for small teams
Runtime security: Falco for detection (CNCF Graduated), Tetragon for eBPF enforcement (<1% overhead)
Supply chain integrity (lessons from Trivy compromise, March 2026)
The Trivy supply chain attack (CVE-2026-33634) is the defining security event of 2026 so far. Attackers force-pushed all GitHub Action tags to credential-stealing malware and published malicious binaries to Docker Hub. Key takeaways:
Pin GitHub Actions to commit SHAs, never mutable tags.uses: aquasecurity/trivy-action@<sha>, not @v0.35.0. Applies to ALL actions, not just Trivy. See also reviewdog/action-setup (CVE-2025-30154), the upstream cause of the tj-actions compromise.
Pin container images to SHA256 digests in CI/CD. Tags can be overwritten; digests cannot.
Monitor for force-push events on action repos you depend on. GitHub's audit log and StepSecurity Harden-Runner can detect this.
Vendor critical CI tools or use pre-built, verified binaries instead of pulling from upstream on every run.
Rotate secrets if any CI pipeline ran compromised Trivy (v0.69.4/5/6) between March 19-23, 2026. The infostealer exfiltrated SSH keys, cloud creds, Docker configs, and k8s tokens.
Trivy safe version: v0.70.0+ for new pins. v0.69.3 was the March 2026 rollback version. Actions such as trivy-action@v0.35.0 and setup-trivy@v0.2.6 still need verified commit SHAs, not mutable tags.
Platform awareness
cgroup v2 required on K8s 1.36+ (FailCgroupV1 defaults to true; kubelet won't start on cgroup v1 unless failCgroupV1: false). Nodes on cgroup v1 (CentOS 7, RHEL 7, Ubuntu 18.04) will fail.
containerd 2.0 required on K8s 1.36+. Last release supporting containerd 1.x is 1.35.
K8s 1.36 launches April 22, 2026 - containerd 2.0 required on all nodes. IPVS kube-proxy mode removal not yet committed to a specific version (nftables is the replacement).
AppArmor annotation auto-population stopped in 1.34; full removal in 1.36. Use securityContext.appArmorProfile field.
DRA (Dynamic Resource Allocation) GA in 1.34 for GPU/FPGA/hardware scheduling. Replaces device plugin model.
User namespaces (hostUsers: false) enabled by default since K8s 1.33. Maps container UID 0 to unprivileged host UID. Huge for PCI multi-tenancy - container breakout doesn't yield host root.
Pod-level mTLS (KEP-4317) beta in 1.35, feature gate PodCertificates must be manually enabled. Native X.509 certs for pods without service mesh. Future alternative to Istio/Cilium for Req 4 compliance.
Compliance
Read references/compliance.md for the full PCI-DSS 4.0 requirements mapping to Kubernetes controls.
Quick reference: PCI-DSS 4.0 on K8s
PCI-DSS 4.0 is the only active version (3.2.1 retired March 2024). 51 future-dated requirements became mandatory March 31, 2025.
CDE isolation: dedicated cluster strongly preferred. Shared cluster puts the entire cluster in PCI scope and requires dedicated node pools + taints, gVisor/Kata for CDE pods, separate DNS, separate audit streams, and extensive QSA documentation. Most QSAs push back on shared clusters.
PCI MPoC: MPoC backends (attestation/monitoring for tap-to-pay) fall under full PCI-DSS scope. No K8s-specific addenda - standard PCI-DSS 4.0 controls apply.
Production Checklist
Manifests
Resource requests AND limits set on every container
All three probes configured (startup, liveness, readiness)
Pinned image tag or SHA256 digest (never :latest)
Security context at pod AND container level: non-root, read-only rootfs, drop ALL caps, seccomp RuntimeDefault
Replicas >= 2 for HA (>= 3 preferred)
topologySpreadConstraints for zone distribution; soft anti-affinity for node spread
Rolling update with maxUnavailable: 0
Standard app.kubernetes.io/* labels
Namespace specified explicitly
Secrets via External Secrets Operator or Sealed Secrets (not in manifests, ConfigMaps, or env vars)
PodDisruptionBudget for HA workloads
terminationGracePeriodSeconds matches app shutdown time
Gateway API HTTPRoute for external access (not legacy Ingress)
Images signed with cosign, verified at admission
Helm
All dependency versions pinned in Chart.yaml
OCI registry for chart distribution (digest-pinned in prod)
All values documented with comments in values.yaml
values.schema.json for input validation
No :latest tags in default values
Resources set in default values
NOTES.txt with post-install instructions
helm template renders clean YAML
Separate values files per environment
.helmignore excludes test/ci artifacts
All hooks have helm.sh/hook-delete-policy
No secrets in Helm values (use ESO/sealed-secrets references)
Architecture
Cluster topology matches scale and isolation needs
GitOps tool chosen with clear promotion strategy (no auto-sync to prod)
Gateway API for external traffic (not legacy Ingress)
Network policies default-deny in all namespaces
Pod Security Standards: enforce: restricted on all app namespaces
ValidatingAdmissionPolicy or Kyverno for custom admission rules
RBAC follows least-privilege; OIDC + MFA for API access
Secrets via ESO + cloud KMS, Vault, or Sealed Secrets (match tool to environment - see Architecture reference)
Images signed (cosign/Sigstore) and verified at admission
CDE in dedicated cluster or hard-isolated with dedicated node pools
etcd encryption via KMS v2 (not disk-level alone)
mTLS between all CDE services (Istio strict / Cilium)
K8s audit logging at RequestResponse level for CDE namespaces
Audit logs shipped to immutable SIEM, automated review rules
SBOMs generated and stored for every image
No hardcoded secrets anywhere (Req 8.6.2)
MFA on all CDE access paths (Req 8.4.2)
WAF on public-facing web apps (Req 6.4.2)
Certificate inventory maintained (Req 4.2.1.1)
Quarterly authenticated internal vulnerability scans (Req 11.3.1.2) - application-level, not just image scanning
Reference Files
references/manifest-templates.md - manifest templates and reusable workload patterns
references/architecture.md - cluster and platform design guidance
references/sealed-secrets.md - Sealed Secrets patterns and caveats
references/compliance.md - PCI-DSS and platform hardening guidance
references/gitops-emergency-changes.md - safe workflow for urgent changes to GitOps-managed workloads
Output Contract
See references/output-contract.md for the full contract.
Skill name: KUBERNETES
Deliverable bucket:audits
Mode: conditional. When invoked to analyze, review, audit, or improve existing repo content, emit the full contract - boxed inline header, body summary inline plus per-finding detail in the deliverable file, boxed conclusion, conclusion table - and write the deliverable to docs/local/audits/kubernetes/<YYYY-MM-DD>-<slug>.md. When invoked to answer a question, teach a concept, build a new artifact, or generate content, respond freely without the contract.
Severity scale:P0 | P1 | P2 | P3 | info (see shared contract; only used in audit/review mode).
Related Skills
docker - for Dockerfile and Compose patterns. Kubernetes deploys the images Docker builds. Image optimization belongs in docker; manifest design belongs here.
ci-cd - for pipeline design that deploys to K8s. Kubernetes skill covers manifests and Helm charts; ci-cd covers the pipeline stages that apply them.
terraform - for provisioning the cluster itself (EKS, GKE, AKS, bare-metal node pools). Terraform creates the cluster; kubernetes configures what runs on it.
cluster-health - for read-only cluster status checks, node/workload diagnostics, events, ingress/storage/log sweeps, and post-maintenance reports.
databases - for deploying databases on K8s (StatefulSets, operators, PVCs). Kubernetes owns the manifest pattern; databases owns the engine configuration within.
ansible - can deploy to K8s via kubernetes.core collection, but manifest and Helm chart design belong here.
Rules
These are non-negotiable. Violating any of these is a bug.
No :latest tags. Pin images to a specific version or SHA256 digest.
Namespace everything. The default namespace is a code smell.
Resource requests AND limits on every pod. No exceptions.
Verify kube context before running any kubectl/helm/argocd command.
Verify requester authorization before cluster changes. In shared chats, do not run kubectl, Helm, ArgoCD, or GitOps edits for admin requests from a non-admin participant. Stop and ask the authorized owner for explicit approval.
No auto-sync to prod. Manual approval or PR-based promotion.
helm template before every apply. Catch template errors before they hit the cluster.
Secrets never in plaintext. Not in Git, not in ConfigMaps, not in Helm values, not in env vars in manifests.
Test changes in staging first. Policy changes, admission controllers, upgrades, SSA migration.
Separate values files per environment. Don't modify values.yaml for env-specific config.
Gateway API for new external access. Ingress-NGINX retired March 2026. Stop deploying new Ingress resources.
Sign images with cosign. Verify at admission. SLSA Level 2 minimum for production.
Run the AI self-check. Every generated manifest gets verified against the checklist above before returning.
Understand resource metric semantics. HPA CPU target is percentage of CPU request, not node capacity. Example: a pod requesting cpu: 100m with averageUtilization: 70 scales when per-pod CPU usage hits 70m (100m * 70%) - it does not matter whether the node has 2 or 64 cores. Don't confuse requests (scheduling floor), limits (enforcement ceiling), and actual usage (what the container is consuming right now).
