Production-grade Kubernetes operations for AI agents. Generates secure, scalable, multi-cloud K8s manifests and Helm charts with built-in failure-mode prevention, compliance validation, and structured output contracts.
Core Workflow — 7-Step Failure-Mode Prevention
Every K8s response MUST follow this sequence. Skip a step → risk a failure mode.
These are the failure modes this skill prevents. Every manifest review and generation MUST check all eight.
FM-1: Insecure Workloads
Symptom: Container running as root, privileged mode, hostPath mounts, no securityContext.
Prevention:runAsNonRoot: true, readOnlyRootFilesystem: true, drop ALL capabilities, add only required ones.
Detection:kubectl get pods -o json | jq '.items[].spec.containers[].securityContext'
FM-2: Resource Starvation
Symptom: No requests/limits, unbounded memory growth, CPU throttling, OOMKilled.
Prevention: Always set requests = limits for Guaranteed QoS on critical workloads. Use LimitRange at namespace level.
Detection:kubectl top pods --namespace=<ns> and check for OOMKilled in pod status.
FM-3: Network Exposure
Symptom: Missing NetworkPolicies, services exposed as LoadBalancer unnecessarily, no TLS termination.
Prevention: Deny-all ingress by default, explicit NetworkPolicy allowlists, TLS via cert-manager.
Detection:kubectl get netpol --all-namespaces and kubectl get svc --all-namespaces | grep LoadBalancer
FM-4: Privilege Sprawl
Symptom: ClusterRoleBindings to cluster-admin, overly broad RBAC, service accounts with secrets access.
Prevention: Least-privilege RBAC, per-namespace Roles, service account token audiences restricted.
Detection:kubectl get clusterrolebindings -o json | jq '.items[] | select(.roleRef.name=="cluster-admin")'
FM-5: Fragile Rollouts
Symptom: No health probes, no PDB, rolling update with single replica, no revision history.
Prevention: Readiness + liveness probes, PodDisruptionBudget, revisionHistoryLimit: 10, minReadySeconds.
Detection:kubectl get deployment -o json | jq '.items[] | select(.spec.replicas==1 and .spec.strategy.type=="RollingUpdate")'
FM-6: API Drift
Symptom: Deprecated extensions/v1beta1, apps/v1beta2, policy/v1beta1 PDB.
Prevention: Always check kubectl api-resources for current API version. Use apps/v1 for Deployments, policy/v1 for PDB.
Detection:kubectl get --raw /apis | jq -r '.groups[].preferredVersion.groupVersion'
FM-7: GitOps Divergence — NEW
Symptom: Manual kubectl apply bypasses GitOps pipeline, cluster state drifts from Git, unrecorded changes.
Prevention: All changes through Git → Flux/ArgoCD reconciliation. kubectl for read-only and emergencies only.
Detection:flux get kustomizations -A or argocd app diff <app> to detect drift.
FM-8: Multi-Cloud Skew — NEW
Symptom: EKS-specific annotations on GKE, AKS ingress class mismatch, OpenShift SCC violations.
Prevention: Platform-conditional manifest generation. Check cloud provider before emitting manifests.
Detection: Validate against provider-specific kubectl api-resources and admission webhooks.
Set min/max on node groups, not on individual deployments
Use cluster-autoscaler.kubernetes.io/safe-to-evict: "true" on pods that can move
Apply PDBs so autoscaler respects availability during scale-down
Workload Rightsizing Rules
Never run without limits. Use LimitRange to enforce defaults
Burstable for dev/staging. Guaranteed for production databases and stateful sets
Preemptible nodes for batch. Tolerations + nodeSelectors
HPA over static replicas. Let metrics drive scaling decisions
Observability
Prometheus Metrics
# Pod annotations for Prometheus scrapingmetadata:annotations:prometheus.io/scrape:"true"prometheus.io/port:"8080"prometheus.io/path:"/metrics"# ServiceMonitor (Prometheus Operator)apiVersion:monitoring.coreos.com/v1kind:ServiceMonitormetadata:name:myappspec:selector:matchLabels:app:myappendpoints:-port:metricsinterval:30spath:/metrics
Structured Logging
# Sidecar or daemonset pattern — emit JSON to stdout# Containers should log to stdout/stderr in JSON format:# {"level":"info","ts":"2026-06-18T02:06:00Z","msg":"request","method":"GET","path":"/api","duration_ms":42,"status":200}# Use Fluent Bit or Vector as DaemonSet for log collection:apiVersion:apps/v1kind:DaemonSetmetadata:name:fluent-bitnamespace:loggingspec:selector:matchLabels:app:fluent-bittemplate:metadata:labels:app:fluent-bitspec:serviceAccountName:fluent-bitcontainers:-name:fluent-bitimage:fluent/fluent-bit:3.1volumeMounts:-name:varlogmountPath:/var/log-name:varlibdockercontainersmountPath:/var/lib/docker/containersreadOnly:true
OpenTelemetry
# Instrument with OTel SDK, configure via OTEL_EXPORTER_OTLP_ENDPOINTenv:-name:OTEL_EXPORTER_OTLP_ENDPOINTvalue:"http://otel-collector.observability:4317"-name:OTEL_SERVICE_NAMEvalue:"myapp"-name:OTEL_RESOURCE_ATTRIBUTESvalue:"deployment.environment=production,cloud.provider=aws"# OpenTelemetry Collector — sidecar pattern# OR use the OpenTelemetry Operator for auto-instrumentation
Golden Signals Dashboard (Grafana / Datadog / New Relic)
Latency: P50, P95, P99 of request duration
Traffic: Requests per second
Errors: 5xx rate + error budget burn rate
Saturation: CPU throttle %, memory pressure, goroutine count
Alert Rules (Prometheus)
groups:-name:apprules:-alert:HighErrorRateexpr:rate(http_requests_total{status=~"5.."}[5m])/rate(http_requests_total[5m])>0.01for:5mlabels:severity:criticalannotations:summary:"Error rate > 1% for {{ $labels.app }}"-alert:PodRestartingexpr:rate(kube_pod_container_status_restarts_total[15m])>0for:5mlabels:severity:warning
Secret Management
External Secrets Operator (ESO)
apiVersion:external-secrets.io/v1beta1kind:ExternalSecretmetadata:name:myapp-secretsspec:refreshInterval:1hsecretStoreRef:name:aws-secretsmanager# or gcp-secretmanager, azure-keyvaultkind:ClusterSecretStoretarget:name:myapp-secretscreationPolicy:Ownerdata:-secretKey:DATABASE_URLremoteRef:key:prod/myapp/database-url-secretKey:API_KEYremoteRef:key:prod/myapp/api-key
### Assumptions- Target cluster: EKS 1.30, namespace: `production`- You have cert-manager and AWS Load Balancer Controller installed
- IRSA is configured for service account IAM roles
### Tradeoffs- Chose Guaranteed QoS for the database pod (slight over-provision, but predictable performance)
- Used Burstable QoS for the API (cost-effective, acceptable for stateless workloads)
- Disabled privilege escalation even though it breaks some debugging tools — security over convenience
### Rollback
\`\`\`bash
kubectl delete -f manifest.yaml
# OR for Helm:
helm uninstall myapp -n production
# OR to undo last rollout:
kubectl rollout undo deployment/myapp -n production
\`\`\`
DO / DON'T
✅ DO
# DO: Complete security contextsecurityContext:runAsNonRoot:truerunAsUser:1000seccompProfile:type:RuntimeDefaultcapabilities:drop: ["ALL"]
readOnlyRootFilesystem:trueallowPrivilegeEscalation:false# DO: Both probeslivenessProbe:httpGet:path:/healthzport:8080readinessProbe:httpGet:path:/readyport:8080# DO: Resource requests AND limitsresources:requests:cpu:100mmemory:128Milimits:cpu:500mmemory:256Mi# DO: Namespace labels for PSSmetadata:labels:pod-security.kubernetes.io/enforce:restricted# DO: Use current API versionsapiVersion:apps/v1apiVersion:networking.k8s.io/v1apiVersion:policy/v1# DO: Pod anti-affinity for HAaffinity:podAntiAffinity:requiredDuringSchedulingIgnoredDuringExecution:-labelSelector:matchLabels:app:myapptopologyKey:kubernetes.io/hostname# DO: Topology spread for zone HAtopologySpreadConstraints:-maxSkew:1topologyKey:topology.kubernetes.io/zonewhenUnsatisfiable:ScheduleAnyway# DO: Provide PDBapiVersion:policy/v1kind:PodDisruptionBudgetspec:minAvailable:1
❌ DON'T
# DON'T: No security contextspec:containers:-name:appimage:myapp:latest# securityContext is MISSING — FM-1# DON'T: Running as rootsecurityContext:runAsUser:0# NO — FM-1# DON'T: Privileged containersecurityContext:privileged:true# NO — FM-1, NEVER in production# DON'T: hostPath mounts without extreme cautionvolumes:-name:dangeroushostPath:path:/var/run/docker.sock# NO — FM-1, container escape risk# DON'T: No resource limitsresources: {} # NO — FM-2# DON'T: No health probes — FM-5# No livenessProbe or readinessProbe defined# DON'T: Deprecated API versions — FM-6apiVersion:extensions/v1beta1# DEPRECATED since 1.16# DON'T: Single replica without PDB — FM-5spec:replicas:1# No PodDisruptionBudget defined# DON'T: Overly broad RBAC — FM-4rules:-apiGroups: ["*"]
resources: ["*"]
verbs: ["*"] # NO — just use cluster-admin if you need this# DON'T: No NetworkPolicy — FM-3# Default deny NetworkPolicy must exist in every namespace# DON'T: Hardcoded cloud provider specifics — FM-8annotations:eks.amazonaws.com/role-arn:"..."# OK if target is known EKS, NOT OK as generic manifest# DON'T: kubectl apply for GitOps-managed resources — FM-7# Use Git → Flux/ArgoCD path instead
Quick Reference Cards
Minimal Production-Ready Deployment Checklist
securityContext with runAsNonRoot: true, allowPrivilegeEscalation: false, capabilities drop ALL
readinessProbe AND livenessProbe defined
resources.requests AND resources.limits set for all containers
HPA with minReplicas >= 2 for stateless workloads
PodAntiAffinity or TopologySpreadConstraints for HA
PodDisruptionBudget with minAvailable: 1 or maxUnavailable: 1
ServiceAccount with cloud IAM annotations (if cloud managed)
NetworkPolicy — deny-all + explicit allowlist
PSS namespace labels (enforce: restricted)
No deprecated API versions (kubectl api-resources verified)
Secrets externalized (ESO/SealedSecrets/Vault), NOT in plain ConfigMap
Prometheus scrape annotations or ServiceMonitor
Troubleshooting Quick Commands
# Pod won't start
kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns> --previous
# Resource issues
kubectl top pods -n <ns>
kubectl get events -n <ns> --sort-by='.lastTimestamp'# RBAC issues
kubectl auth can-i --list --as=system:serviceaccount:<ns>:<sa> -n <ns>
# Network issues
kubectl run tmp --rm -it --image=nicolaka/netshoot -n <ns> -- /bin/bash
# Then: curl, nc, dig, tcpdump from inside cluster# Drift detection (Flux)
flux get kustomizations -A --status-selector ready=false# Drift detection (ArgoCD)
argocd app diff <app>
# API version check
kubectl api-resources --verbs=list -o wide
kubectl explain deployment --api-version=apps/v1
Evaluation
See evals/eval_cases.json for trigger match test cases and near-miss negatives.
References
references/k8s-security-hardening.md — Full security reference: PSS, RBAC, NetworkPolicies, OWASP Top 10
references/k8s-failure-modes.md — All 8 failure modes with detection and remediation
