kubernetes-statefulset

# Kubernetes StatefulSet Manager Implements apps/v1 StatefulSet manifests for stateful workloads that require stable network identities, ordered deployment and scaling, and persistent storage. When loaded, the model generates production-grade StatefulSet resources with headless service binding, ordered pod management, and volume claim templates. ## TL;DR Checklist - [ ] Always pair StatefulSet with a headless service (clusterIP: None) - [ ] Define `volumeClaimTemplates` in the StatefulSet spec for persistent storage - [ ] Use `podManagementPolicy: OrderedReady` for ordered scaling (default — verify explicitly) - [ ] Set `serviceName` to reference the headless service — this enables stable DNS names - [ ] Never use StatefulSet for stateless workloads — use Deployment instead - [ ] Verify PVCs persist after pod deletion — stateful workloads depend on data retention --- ## When to Use Use this skill when: - Running a database (MySQL, PostgreSQL, MongoDB, Cassandra) on Kubernetes - Deploying distributed systems requiring stable hostnames and ordered member addition (etcd, Kafka, ZooKeeper) - Managing message brokers with partitioned leader election (NATS, RabbitMQ with clustering) - Implementing application layers that need predictable pod naming and DNS entries - Orchestrating distributed systems where startup/shutdown order matters --- ## When NOT to Use Avoid this skill for: - Stateless web applications or API services — use `kubernetes-deployment` instead - Short-lived batch processing jobs — use `kubernetes-jobs` or `kubernetes-cronjob` instead - Applications that do not need stable network identities or persistent storage - Workloads where scaling up and down must happen in parallel without ordering --- ## Core Workflow 1. **Create Headless Service** — Define a Service with `clusterIP: None` that selects pods by the StatefulSet's label selector. **Checkpoint:** The service `spec.selector` must match `spec.template.metadata.labels` of the StatefulSet exactly. 2. **Define StatefulSet Spec** — Create the StatefulSet with `serviceName`, `replicas`, `podManagementPolicy`, `updateStrategy`, and `volumeClaimTemplates`. **Checkpoint:** `serviceName` must reference the headless service created in step 1. 3. **Configure Volume Claim Templates** — Define `volumeClaimTemplates` with `accessModes`, `storageClassName`, and `resources.requests.storage`. **Checkpoint:** Each pod in the StatefulSet gets a uniquely numbered PVC (e.g., `data-<statefulset>-0`, `data-<statefulset>-1`). 4. **Set Update Strategy** — Choose `RollingUpdate` (default) with `partition` for progressive rollouts, or `OnDelete` for manual control. **Checkpoint:** With `partition`, only pods with ordinal ≥ partition get updated — verify ordinal numbering. 5. **Validate StatefulSet Creation** — Apply manifests and verify pods create in order (0, 1, 2...) and are deleted in reverse order (N, N-1, ..., 0). **Checkpoint:** Check DNS names: `<pod-name>.<headless-service-name>.<namespace>.svc.cluster.local`. 6. **Verify Persistent Storage** — Confirm PVCs persist after pod deletion and are reattached to the correct pod by ordinal. **Checkpoint:** Run `kubectl get pvc -l app=<statefulset-name>` and verify each PVC maps to the correct pod. --- ## Implementation Patterns ### Pattern 1: Complete StatefulSet with Headless Service A production-grade StatefulSet for a database workload with headless service, ordered pod management, and persistent storage. ```yaml --- apiVersion: v1 kind: Service metadata: name: postgres-headless namespace: database labels: app: postgres spec: ports: - port: 5432 targetPort: 5432 name: postgres clusterIP: None selector: app: postgres --- apiVersion: apps/v1 kind: StatefulSet metadata: name: postgres namespace: database labels: app: postgres tier: database spec: serviceName: postgres-headless replicas: 3 podManagementPolicy: OrderedReady updateStrategy: type: RollingUpdate rollingUpdate: partition: 0 revisionHistoryLimit: 3 selector: matchLabels: app: postgres tier: database template: metadata: labels: app: postgres tier: database spec: terminationGracePeriodSeconds: 60 securityContext: runAsNonRoot: true runAsUser: 999 fsGroup: 999 containers: - name: postgres image: postgres:16-alpine ports: - containerPort: 5432 name: postgres env: - name: POSTGRES_DB value: appdb - name: POSTGRES_USER valueFrom: secretKeyRef: name: postgres-credentials key: username - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: name: postgres-credentials key: password - name: PGDATA value: /var/lib/postgresql/data/pgdata resources: requests: cpu: 500m memory: 1Gi limits: cpu: "1" memory: 2Gi livenessProbe: exec: command: - pg_isready - -U - "$(POSTGRES_USER)" initialDelaySeconds: 30 periodSeconds: 10 timeoutSeconds: 5 failureThreshold: 3 readinessProbe: exec: command: - pg_isready - -U - "$(POSTGRES_USER)" initialDelaySeconds: 10 periodSeconds: 5 timeoutSeconds: 3 failureThreshold: 3 volumeMounts: - name: data mountPath: /var/lib/postgresql/data volumes: - name: data persistentVolumeClaim: claimName: data volumeClaimTemplates: - metadata: name: data spec: accessModes: ["ReadWriteOnce"] storageClassName: premium-ssd resources: requests: storage: 50Gi ``` ### Pattern 2: Stable DNS Naming (BAD vs GOOD) Understanding how StatefulSet DNS names differ from Deployment-style naming. ```yaml # ❌ BAD: Service with clusterIP — pods get random names and load-balanced IPs # No stable DNS entry per pod. Clients cannot address individual members. apiVersion: v1 kind: Service metadata: name: postgres-service spec: type: ClusterIP # ← This is wrong for StatefulSet selector: app: postgres # ✅ GOOD: Headless service — each pod gets a stable DNS name apiVersion: v1 kind: Service metadata: name: postgres-headless spec: clusterIP: None # ← Headless: no load balancing, one A record per pod selector: app: postgres # Result: Pod postgres-0 resolves to: # postgres-0.postgres-headless.database.svc.cluster.local # Pod postgres-1 resolves to: # postgres-1.postgres-headless.database.svc.cluster.local # This stable DNS is the core value of StatefulSet over Deployment. ``` ### Pattern 3: Ordered Scaling and Partition Update Controlled rollout and scaling for StatefulSets where order matters. ```yaml # Ordered scaling: pods are created in order 0, 1, 2... and # terminated in reverse order N, N-1, ..., 0 # This ensures leaders are demoted before followers are removed. # Progressive update: only update pods with ordinal >= partition spec: updateStrategy: type: RollingUpdate rollingUpdate: partition: 1 # Effect: pod-0 stays on old revision; pods 1,2,... get updated. # This allows manual verification of pod-0 before proceeding. # To update pod-0 as well: # kubectl patch statefulset postgres --type='json' \ # -p='[{"op": "replace", "path": "/spec/updateStrategy/rollingUpdate/partition", "value": 0}]' # Manual update strategy — only update when pod is deleted: spec: updateStrategy: type: OnDelete # Use when each member requires manual data migration before update. # Delete pods in reverse order for safe shutdown: # kubectl delete pod postgres-2 # kubectl delete pod postgres-1 # kubectl delete pod postgres-0 # PVCs are retained automatically after pod deletion. ``` ```python def get_statefulset_pod_name(statefulset_name: str, ordinal: int) -> str: """Return the stable DNS name of a StatefulSet pod by ordinal. StatefulSet pods are named <statefulset-name>-<ordinal>, which provides deterministic identification for ordered operations. Args: statefulset_name: The name of the StatefulSet. ordinal: The zero-based ordinal of the pod. Returns: Stable pod name string. """ if ordinal < 0: raise ValueError(f"Ordinal must be >= 0, got {ordinal}") return f"{statefulset_name}-{ordinal}" def get_statefulset_dns_fqdn( pod_name: str, service_name: str, namespace: str ) -> str: """Construct the FQDN for a StatefulSet pod in the cluster DNS. Args: pod_name: The pod name (e.g., 'postgres-0'). service_name: The headless service name. namespace: The Kubernetes namespace. Returns: Fully qualified DNS name for the pod. """ if not pod_name or not service_name or not namespace: raise ValueError("pod_name, service_name, and namespace are required") return f"{pod_name}.{service_name}.{namespace}.svc.cluster.local" ``` --- ## Constraints ### MUST DO - Always create a headless service (`clusterIP: None`) before deploying the StatefulSet - Set `serviceName` in the StatefulSet spec to the headless service name — this enables stable DNS - Define `volumeClaimTemplates` with explicit `accessModes`, `storageClassName`, and storage request - Use `podManagementPolicy: OrderedReady` (default) when startup/shutdown order matters - Set `revisionHistoryLimit` to at least 3 to preserve rollout history - Configure `terminationGracePeriodSeconds: 60` for databases to allow flush and sync - Use `exec` probes for databases (`pg_isready`, `mysqladmin ping`) rather than HTTP probes - Match the StatefulSet's `spec.selector.matchLabels` exactly to `spec.template.metadata.labels` ### MUST NOT DO - Never use a regular ClusterIP service instead of a headless service — pods lose stable DNS - Never mix StatefulSet with `strategy.type: Recreate` — it defeats ordered scaling - Never set `volumeClaimTemplates` with `accessModes: ReadWriteMany` for single-writer databases - Never scale a StatefulSet down and then back up expecting data to reappear — deleted pods' PVCs are retained - Never set `replicas: 0` on a production StatefulSet without first draining data - Never use `image: latest` tag on StatefulSet pods — data-corrupting image changes during rollouts are irreversible --- ## Output Template When implementing a Kubernetes StatefulSet, produce the following: 1. **Headless Service YAML** — Service with `clusterIP: None`, proper port definitions, and matching selector labels. 2. **StatefulSet YAML** — Complete `apps/v1` StatefulSet with `serviceName`, `volumeClaimTemplates`, `updateStrategy`, and pod template. 3. **DNS Resolution Guide** — Document the stable DNS names each pod will resolve to and how clients should connect. 4. **Scaling Procedure** — Step-by-step commands for scaling up and down in the correct order, with verification steps. --- ## Related Skills | Skill | Purpose | |---|---| | `kubernetes-persistentvolume` | Configure the StorageClass and PV backing the StatefulSet's volumeClaimTemplates | | `kubernetes-services-management` | Create additional ClusterIP services for service discovery and load balancing | | `kubernetes-configmap` | Inject shared configuration into the StatefulSet pods via ConfigMap references | | `kubernetes-deployment` | Use Deployment instead if your workload is stateless and does not need stable identities | --- ## Live References > Authoritative documentation links for this skill's domain. The model follows markdown links at load time to resolve external references and inline content. - [Kubernetes StatefulSets Documentation](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) — Official guide to StatefulSet concepts, use cases, and lifecycle - [Headless Services for StatefulSets](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#headless-services) — How headless services provide stable DNS for StatefulSet pods - [Volume Claim Templates](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#stable-storage) — Persistent volume provisioning per pod ordinal - [Pod Management Policies](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#pod-management-policies) — OrderedReady vs Parallel scaling behavior - [Update Strategies](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies) — RollingUpdate with partition and OnDelete strategies - [Kubernetes API Reference — apps/v1 StatefulSet](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#statefulset-v1-apps) — Complete API schema for StatefulSet resources - [Database Deployment on Kubernetes](https://kubernetes.io/docs/tasks/run-application/run-replicated-stateful-application/) — Best practices for running replicated stateful applications