kubernetes-security

name

kubernetes-security

description

Harden a Kubernetes cluster's data plane and control plane. Covers Pod Security Standards (Restricted, Baseline, Privileged), RBAC with least privilege, NetworkPolicy default-deny, secrets management without raw env vars, admission controllers (Kyverno, OPA Gatekeeper), image scanning, and audit logging. Invoke when provisioning a new cluster, inheriting one, or before adding a new tenant to a shared cluster.

Kubernetes Security

A pragmatic baseline for a single Kubernetes cluster running a small-team workload. Skews toward "I have a cluster and need it to not be the cause of an incident" — not a full CIS Benchmark for regulated environments. Most managed-K8s providers ship sensible defaults at the control-plane layer; the data-plane (your workloads) is where the work is.

When to invoke

• Provisioning a new cluster
• Inheriting a cluster with no documented hardening
• Before adding a new tenant / team / workload to a shared cluster
• After a K8s security advisory affecting your version
• Periodic re-audit (quarterly)
• Multi-tenant SaaS where customer workloads share infrastructure

Step 1 — Cluster baseline

Before workload hardening, the cluster itself.

# What version? Is it supported?
kubectl version --short
# K8s has a ~14-month support window. Unsupported = unpatched CVEs.

# Who can reach the API?
kubectl cluster-info
# For managed clusters, check the provider's "API endpoint" access setting:
# - Private endpoint, restricted CIDRs, or behind a bastion is correct
# - "Public, open to 0.0.0.0/0" is a finding

# Audit log enabled?
# For EKS/GKE/AKS: check the cluster's "audit logging" / "control-plane logging" setting
# Self-hosted: --audit-log-path on the API server

Patterns:

• Restrict the API endpoint. Public + open is the worst default. Use private endpoint + bastion, or public + CIDR allowlist + IAM auth at minimum.
• Enable control-plane audit logging. Without it, an incident has no record. Ship to a log destination outside the cluster (so a compromise cannot wipe its own audit trail).
• Pin to a supported version, automate patching to within the support window.
• Encrypt etcd at rest. Default-on for EKS, GKE, AKS. Self-hosted: configure --encryption-provider-config.

Step 2 — Pod Security Standards

Since K8s 1.25, Pod Security Admission is built-in. There are three policy levels:

Level	What it allows	When to use
Privileged	Everything	System namespaces only (kube-system, ingress controllers that need it)
Baseline	No privileged escalation, no host networking, no hostPath	Default for most workloads
Restricted	The above plus: non-root user, read-only root filesystem, drop all capabilities, seccomp profile	Production application workloads

Apply per namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted

enforce rejects non-compliant pods. warn returns a warning to the user. audit logs to the audit log.

For workloads that cannot meet restricted (legacy containers needing root), use baseline and document why. Anything in privileged mode is a finding unless it's a known infrastructure component.

Common adjustments to make existing workloads restricted-compliant:

# In the pod spec
securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  runAsGroup: 1000
  fsGroup: 1000
  seccompProfile:
    type: RuntimeDefault

# Per container
containers:
- name: app
  securityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop: [ALL]
    runAsNonRoot: true

Anything writable (cache, temp) goes to an emptyDir volume; the root filesystem stays read-only.

Step 3 — RBAC and ServiceAccounts

The single most common K8s misconfiguration: every workload running as the default ServiceAccount of its namespace, which often has more rights than the workload needs.

# What can the default ServiceAccount in 'production' do?
kubectl auth can-i --list --as=system:serviceaccount:production:default -n production

# What ServiceAccounts have cluster-admin or near-admin rights?
kubectl get clusterrolebindings -o json \
  | jq -r '.items[] | select(.roleRef.name == "cluster-admin") | "\(.metadata.name) -> \(.subjects | map(.name) | join(","))"'

Patterns:

• One ServiceAccount per workload. Don't use default.
• automountServiceAccountToken: false on workloads that don't talk to the K8s API. Most don't.
• Bind to a Role (namespace-scoped) before a ClusterRole — ClusterRoleBinding should be rare and justified.
• Avoid * verbs and * resources in Role rules.
• Audit system:authenticated and system:anonymous — these should have minimal permissions, never cluster-admin.

Example tight Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  resourceNames: ["app-config"]
  verbs: ["get"]

The resource name pin matters — verbs: [get] on all configmaps is too broad if the workload only reads one.

Step 4 — NetworkPolicy (default-deny)

By default, every pod can talk to every other pod, in any namespace. That's a flat network — one compromised workload pivots freely.

# Default-deny ingress AND egress in a namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Then explicitly allow what's needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: app-allow
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes: [Ingress, Egress]
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-nginx
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: postgres
    ports:
    - protocol: TCP
      port: 5432
  - to:                            # DNS
    - namespaceSelector: {}
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53

NetworkPolicy requires a compatible CNI (Calico, Cilium, Weave, Antrea, AWS VPC CNI with policy enabled). If the CNI ignores NetworkPolicy, the manifests are decoration.

Step 5 — Secrets management

K8s Secret objects are base64-encoded, not encrypted at the application level. At rest in etcd they are encrypted only if you configured encryption. They are still accessible to anyone with get secrets in the namespace.

Patterns, from worst to best:

1. Env vars from kind: Secret — workable but secrets leak via kubectl describe pod, env-dumping logs, child processes
2. Mounted as files via volume — better; not in env, lifecycle is the pod
3. External secret manager via CSI driver / External Secrets Operator — secrets stay in Vault / AWS Secrets Manager / GCP Secret Manager, K8s only sees references
4. Workload Identity / IRSA — workload assumes a cloud IAM role directly, no secret crosses K8s at all

Standard tooling:

• External Secrets Operator (external-secrets.io) — syncs from cloud secret managers
• SealedSecrets (Bitnami) — encrypt secrets in the repo with a cluster-specific key; the controller decrypts at apply time
• SOPS + age — encrypted secrets in Git, decrypted at apply time via a Flux/Argo plugin
• HashiCorp Vault with the Vault Agent injector

For new clusters, prefer External Secrets + a cloud secret manager. SOPS is a good fit for GitOps-heavy teams.

Step 6 — Admission controllers

Policy-as-code that blocks non-compliant resources at API time. Two common choices:

• Kyverno — K8s-native syntax, easier learning curve
• OPA Gatekeeper — Rego policies, more flexible, used at scale

Baseline policies worth enforcing:

• Reject images from non-allowlisted registries
• Reject images tagged :latest (only digests or version tags allowed)
• Reject containers without runAsNonRoot: true (catches workloads that slipped past PSS)
• Reject NetworkPolicy-less namespaces in production
• Require resource limits (CPU/memory) on every container
• Reject hostNetwork: true, hostPID: true, hostIPC: true outside of audited exceptions

# Kyverno example — require resource limits
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resources
spec:
  validationFailureAction: enforce
  rules:
  - name: validate-resources
    match:
      any:
      - resources:
          kinds: [Pod]
    validate:
      message: "Resource limits are required."
      pattern:
        spec:
          containers:
          - resources:
              limits:
                memory: "?*"
                cpu: "?*"

Step 7 — Image hygiene

• Scan every image before it lands in the cluster. Trivy, Grype, or your registry's built-in scanner. See docker-container-security.
• Pull by digest, not tag. myapp@sha256:abc123... is immutable; myapp:v1.2.3 can be repushed.
• Use a private registry for app images; the public Docker Hub is a supply-chain surface.
• Enable registry-side admission that rejects unsigned / unscanned images. Sigstore / cosign for signing; Kyverno or Gatekeeper for enforcement.

Step 8 — Cluster + control-plane hygiene

• kubeconfigs are credentials. Treat them like SSH private keys. Rotate on team change. Scope per-user.
• No long-lived static admin token. Use SSO (cloud provider's OIDC) or short-lived tokens.
• Don't kubectl exec in production casually. It bypasses normal audit logging in some setups. Configure cluster audit policy to capture exec events explicitly.
• kubectl proxy opens the API to localhost without auth — never on a shared dev machine.
• Restrict who can create resources in kube-system — that namespace's pods can compromise the cluster.

Step 9 — Multi-tenancy (if applicable)

If multiple teams or customers share a cluster, hard isolation is hard. Options:

• Namespace per tenant + strict RBAC + NetworkPolicy + ResourceQuotas — minimum bar
• vCluster / Capsule — virtual clusters inside one cluster, more isolated but operational overhead
• Cluster per tenant — strongest isolation, highest cost

A single-cluster multi-tenant setup with shared nodes is soft isolation. Container escape and kernel CVEs can cross tenants. For security-sensitive multi-tenancy, run separate node pools per tenant or move to per-tenant clusters.

Common findings on inherited clusters

Walk this list against any cluster you're auditing:

• API endpoint open to 0.0.0.0/0
• Pod Security Admission not configured (or only at audit, not enforce)
• Workloads using default ServiceAccount
• ClusterRoleBindings to cluster-admin for human accounts or generic ServiceAccounts
• No NetworkPolicy in any namespace (flat network)
• CNI does not support NetworkPolicy (manifests present but ineffective)
• Secrets in env vars throughout
• Images pulled by :latest
• No image scanning
• No admission controller
• Containers running as root (runAsUser: 0 or unspecified)
• privileged: true on application workloads
• hostNetwork: true or hostPath: volumes outside system pods
• Audit logging disabled or sent only to the cluster itself
• kubeconfigs for cluster-admin shared on multiple machines
• etcd backups absent or unencrypted

Quick audit script

#!/usr/bin/env bash
echo "=== Cluster version ==="
kubectl version --short

echo "=== PSS labels per namespace ==="
kubectl get ns -o json | jq -r '.items[] | "\(.metadata.name)\t\(.metadata.labels // {} | to_entries | map(select(.key | startswith("pod-security.kubernetes.io"))) | from_entries)"'

echo "=== ServiceAccounts with cluster-admin ==="
kubectl get clusterrolebindings -o json | jq -r '.items[] | select(.roleRef.name == "cluster-admin") | "\(.metadata.name)\t\(.subjects // [] | map("\(.kind):\(.namespace // "-"):\(.name)") | join(", "))"'

echo "=== Namespaces without NetworkPolicy ==="
for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  count=$(kubectl get networkpolicy -n "$ns" 2>/dev/null | grep -c -v NAME)
  [ "$count" -eq 0 ] && echo "$ns"
done

echo "=== Pods running as root ==="
kubectl get pods -A -o json | jq -r '.items[] | select(.spec.securityContext.runAsNonRoot != true and (.spec.containers[]?.securityContext.runAsNonRoot // false) != true) | "\(.metadata.namespace)/\(.metadata.name)"' | head -30

echo "=== Pods using :latest ==="
kubectl get pods -A -o json | jq -r '.items[] | "\(.metadata.namespace)/\(.metadata.name)\t\(.spec.containers[].image)"' | grep ':latest' | head -30

Checklist

• K8s version is supported; auto-patching within the window
• API endpoint restricted (private or CIDR-allowlisted)
• Control-plane audit logging on, shipped off-cluster
• etcd encryption at rest
• Pod Security Standards: restricted enforced in app namespaces
• Every workload has a dedicated ServiceAccount with minimal RBAC
• automountServiceAccountToken: false for workloads not calling K8s API
• Default-deny NetworkPolicy in every namespace; explicit allows
• CNI supports NetworkPolicy enforcement
• Secrets via External Secrets / SealedSecrets / Vault, not raw kind: Secret in env
• Image scanning before admission
• Images pinned by digest or version, not :latest
• Admission controller (Kyverno / Gatekeeper) enforcing baseline policies
• No privileged, hostNetwork, hostPID, hostIPC in app workloads
• kubeconfigs scoped per user, rotated on offboarding
• Multi-tenancy strategy documented (or single-tenant explicitly)

What this skill will not do

• Replace a full CIS Kubernetes Benchmark for regulated environments
• Help compromise a cluster you do not own
• Recommend privileged: true or hostNetwork: true for application workloads