k8s-hardening-audit

name

k8s-hardening-audit

description

Audit MCP Runtime Kubernetes posture — RBAC, ServiceAccounts, Pod Security Standards, NetworkPolicies, manifest hygiene, secret access, and cluster CIS controls — across operator, sentinel, registry, traefik, and mcp-servers namespaces. Use when Codex reviews or changes RBAC roles, ServiceAccounts, Deployments, NetworkPolicies, PSS labels, securityContext, or k8s manifests under k8s/, config/, or service charts; or when asked for a cluster hardening review.

Kubernetes Hardening Audit

Overview

Use this skill to assess MCP Runtime's cluster posture: RBAC graph, Pod Security Standards, NetworkPolicy enforcement, manifest hygiene, and CIS benchmark compliance. Findings use the shared template at ../_shared/FINDINGS-TEMPLATE.md.

For runtime authn/authz, gateway policy, and protocol fuzzing, use security-audit-platform. For images and supply chain, use supply-chain-audit.

Step 1 — Inventory

Repo-owned namespaces (per CLAUDE.md):

• mcp-runtime — operator.
• mcp-sentinel — api, ui, ingest, processor, gateway, observability.
• mcp-servers — user MCP server workloads, gateway sidecars.
• registry — Distribution v2 registry.
• traefik — ingress controller (or kube-system/traefik on k3s).

For each:

NS=mcp-sentinel
kubectl -n "$NS" get all,sa,role,rolebinding,networkpolicy -o name
kubectl get clusterrole,clusterrolebinding -o name | grep -iE 'mcp|sentinel'

Record SA → Role/ClusterRole bindings as a graph; this becomes the input to RBAC analysis.

Step 2 — RBAC graph analysis

Install the helpers:

go install github.com/corneliusweig/rakkess/cmd/rakkess@latest
go install github.com/reactiveops/rbac-lookup@latest

Per ServiceAccount:

for sa in $(kubectl -n mcp-sentinel get sa -o name); do
  echo "=== $sa ==="
  rakkess --sa "$(echo $sa | cut -d/ -f2)" --namespace mcp-sentinel
done

Findings to flag:

• * verbs on secrets → High unless explicitly justified by a single controller and scoped to a single namespace.
• Cluster-scoped roles on workload SAs → High; scope to a Role unless the controller genuinely watches cluster-wide.
• pods/exec, pods/portforward, pods/attach → High outside break-glass paths.
• serviceaccounts/token create → High; token impersonation primitive.
• escalate or bind on roles → Critical.
• Bindings to system:authenticated or system:unauthenticated → Critical.

Confirm Traefik narrowing (CLAUDE.md): bundled manifests should watch only registry, mcp-sentinel, mcp-servers. Any extra namespace with a traefik-watch Role binding without a documented reason is a Medium finding.

Step 3 — Pod Security Standards

The repo claims "restricted pod-security labels in repo-owned namespaces." Verify every namespace label:

kubectl get ns mcp-runtime mcp-sentinel mcp-servers registry traefik \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels}{"\n"}{end}'

Each repo-owned namespace should set:

• pod-security.kubernetes.io/enforce=restricted
• pod-security.kubernetes.io/audit=restricted
• pod-security.kubernetes.io/warn=restricted

Any namespace at baseline or privileged is a finding (severity by what runs there: workload namespaces → High, observability namespaces with sidecars that need extra caps → Medium with documentation).

Then run kube-linter against checked-in manifests:

go install golang.stackrox.io/kube-linter/cmd/kube-linter@latest
kube-linter lint k8s/ config/

Triage every check; common ones to confirm pass on every Deployment:

• run-as-non-root
• read-only-root-filesystem
• drop-net-raw-capability and drop-all-capabilities
• no-host-network, no-host-pid, no-host-ipc
• privileged-container absent
• seccomp-profile-set (RuntimeDefault)
• cpu-requests, memory-requests, memory-limits present
• liveness-probe and readiness-probe defined
• image-tag-not-latest (tag is a digest or stable version, not latest)

Each violation is a finding. Severity: High for missing non-root and read-only root FS on internet-facing pods; Medium for missing seccomp or limits; Low for missing probes on jobs.

Step 4 — NetworkPolicy audit

kubectl get networkpolicy -A

For each namespace that holds workloads, confirm:

• A default-deny ingress policy exists.
• A default-deny egress policy exists or egress is explicitly allowed only to documented destinations (DNS, ingest service, k8s API).
• The mcp-server gateway sidecar can reach the ingest endpoint, the k8s API (for SAR/auth), and the user MCP container — and nothing else.
• User MCP server containers cannot egress to mcp-sentinel directly, only via the sidecar.
• The registry namespace egress is limited (no random outbound).

Probe with a test pod:

kubectl -n mcp-servers run probe --rm -i --image=busybox:1.36 --restart=Never -- sh -c '
  wget -T2 -qO- http://mcp-platform-api.mcp-sentinel.svc:8080/health || echo BLOCKED
'

Successful unauthorized cross-namespace traffic where the design says it should be blocked is High.

Step 5 — Manifest hygiene per Deployment / StatefulSet

For every workload manifest in k8s/ and config/:

• automountServiceAccountToken: false unless the workload calls the k8s API.
• securityContext at pod and container level: runAsNonRoot: true, runAsUser ≥ 1000, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities.drop: [ALL], seccompProfile.type: RuntimeDefault.
• Volumes mounted read-only when possible; no hostPath outside dev/test manifests (and when present in k8s/, the file name should make the dev intent obvious — *-hostpath.yaml).
• Resource requests and limits set; CPU limit may be omitted for latency-sensitive components, but flag the absence.
• imagePullPolicy: IfNotPresent for pinned digests; Always only for latest (which itself should be flagged).
• No env blocks containing literal secrets — all secret material via valueFrom.secretKeyRef.

kubectl get deploy,sts -A -o json | jq filters can quickly enumerate violations across the live cluster:

kubectl get deploy,sts -A -o json | jq -r '
  .items[]
  | select(.spec.template.spec.containers[]
           | .securityContext.readOnlyRootFilesystem != true)
  | "\(.kind)/\(.metadata.namespace)/\(.metadata.name)"
'

Repeat with filters for runAsNonRoot, allowPrivilegeEscalation, and automountServiceAccountToken.

Step 6 — Secret access and storage

• Confirm every Secret in mcp-sentinel is mounted by exactly the workloads that need it; no broad secrets get on workload SAs.
• Confirm secrets are not committed to git: gitleaks (handled by supply-chain-audit) and grep -RIn 'kind: Secret' k8s/ config/ — the only tracked Secrets should be 02-secrets.yaml.example (no real values) and the bootstrap job that rotates PLATFORM_ADMIN_PASSWORD.
• Confirm the cluster has encryption-at-rest configured for etcd (cluster-level, often out of scope for MCP Runtime to enforce, but call out as a recommendation in the report).
• Confirm ServiceAccount token volumes use projected tokens with a bounded expirationSeconds (default in modern k8s, but verify).

Step 7 — CIS benchmark (live cluster)

docker run --rm --pid=host -v /etc:/etc:ro -v /var:/var:ro \
  -t docker.io/aquasec/kube-bench:latest run --targets master,node,etcd,policies

Triage [FAIL] lines; many will be cluster-operator concerns (kubelet flags, etcd config). Keep MCP Runtime-relevant ones in the report:

• 5.1.x — RBAC and ServiceAccount hardening.
• 5.2.x — Pod Security Standards.
• 5.3.x — NetworkPolicy and CNI.
• 5.7.x — General policies (default SA, default NetworkPolicy).

For each MCP-Runtime-relevant FAIL, propose the manifest change to fix it.

Step 8 — Admission policies

If the cluster runs Kyverno, OPA Gatekeeper, or Sigstore policy controller:

kubectl get clusterpolicies.kyverno.io -A 2>/dev/null
kubectl get constraints -A 2>/dev/null

Confirm policies enforce the hygiene rules above so a malicious or mis-configured manifest is rejected at admission, not just flagged in CI. Absence of any admission-time enforcement is at least a Low finding for an alpha project, Medium when targeting production.

Step 9 — Report

Use ../_shared/FINDINGS-TEMPLATE.md. In the Summary include:

• Namespaces audited and their PSS levels.
• ServiceAccount count and how many have * or cluster-scoped verbs.
• Deployments scanned and the pass rate per kube-linter rule.
• NetworkPolicy coverage (% of pods covered by a default-deny).
• Notable CIS FAILs that the repo can fix.

Cross-reference findings with security-audit-platform and supply-chain-audit to avoid double counting (e.g., a CVE in the operator image is logged once in supply-chain, not twice).