cassandra-k8s-deploy

name

cassandra-k8s-deploy

description

Deploy and manage production Apache Cassandra clusters on Kubernetes using the K8ssandra operator. Covers EKS, GKE, and AKS with cloud-specific storage classes, monitoring (Prometheus/Grafana), backup/restore (Medusa), automated repairs (Reaper), TLS encryption, authentication, and scaling. Use this skill whenever the user wants to deploy Cassandra on Kubernetes, set up a K8ssandra cluster, configure Cassandra backups on K8s, add monitoring to Cassandra on K8s, scale a Cassandra datacenter, or manage any aspect of Cassandra running in a Kubernetes environment.

Deploying Apache Cassandra on Kubernetes with K8ssandra

This skill guides you through deploying production-grade Apache Cassandra clusters on Kubernetes using the K8ssandra operator. K8ssandra bundles the Cassandra operator (cass-operator), Medusa for backups, Reaper for repairs, and metrics integration into a single declarative workflow.

Workflow Overview

1. Identify the target cloud (EKS, GKE, or AKS) — read the matching reference file
2. Install prerequisites (cert-manager, then k8ssandra-operator)
3. Create cloud-specific StorageClass
4. Write and apply a K8ssandraCluster manifest
5. Configure production features: monitoring, backups, repairs, TLS, auth
6. Verify the deployment

Ask the user which cloud provider they're targeting, then read the appropriate reference file:

• AWS EKS → references/eks.md
• Google GKE → references/gke.md
• Azure AKS → references/aks.md

If the user hasn't specified, ask them before proceeding — storage classes and networking details differ significantly across providers.

Step 1: Install Prerequisites

cert-manager (required by the operator for webhook certificates)

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --set installCRDs=true

Wait for cert-manager pods to be ready before continuing:

kubectl wait --for=condition=Ready pods --all -n cert-manager --timeout=120s

K8ssandra operator

helm repo add k8ssandra https://helm.k8ssandra.io/stable
helm repo update
helm install k8ssandra-operator k8ssandra/k8ssandra-operator \
  -n k8ssandra-operator --create-namespace

For multi-namespace deployments, add --set global.clusterScoped=true.

Step 2: Create a Cloud-Specific StorageClass

The StorageClass must use volumeBindingMode: WaitForFirstConsumer so that PVs are provisioned in the same availability zone as the scheduled pod. Without this, Cassandra pods can fail to mount their volumes.

Read the cloud-specific reference file for the exact StorageClass YAML. Summary:

Cloud	StorageClass name	Provisioner	Disk type
EKS	gp3-csi	ebs.csi.aws.com	gp3
GKE	pd-ssd	pd.csi.storage.gke.io	pd-ssd
AKS	managed-premium	disk.csi.azure.com	Premium_LRS

Step 3: Write the K8ssandraCluster Manifest

Here is a production-ready template. Customize the values marked with comments:

apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
  name: production                    # cluster name
  namespace: k8ssandra-operator
spec:
  cassandra:
    serverVersion: "4.1.3"            # Cassandra version (4.0.x, 4.1.x, or 5.0.x)
    datacenters:
      - metadata:
          name: dc1
        size: 3                       # number of nodes (use multiples of rack count)
        racks:                        # one rack per AZ for fault tolerance
          - name: rack1
            nodeAffinityLabels:
              topology.kubernetes.io/zone: ZONE_A
          - name: rack2
            nodeAffinityLabels:
              topology.kubernetes.io/zone: ZONE_B
          - name: rack3
            nodeAffinityLabels:
              topology.kubernetes.io/zone: ZONE_C
        storageConfig:
          cassandraDataVolumeClaimSpec:
            storageClassName: STORAGE_CLASS  # from Step 2
            accessModes: [ReadWriteOnce]
            resources:
              requests:
                storage: 100Gi        # size per node
        resources:
          requests:
            cpu: 2000m
            memory: 8Gi
          limits:
            cpu: 4000m
            memory: 12Gi
        config:
          jvmOptions:
            heapSize: 4G             # ~50% of memory limit, max 8G
          cassandraYaml:
            num_tokens: 16
        softPodAntiAffinity: true    # spread pods across nodes
  # --- Production features (see sections below) ---
  reaper:
    autoScheduling:
      enabled: true
    keyspace: reaper_db

Replace the ZONE_* placeholders with actual availability zones from the cloud reference file. Set size to a multiple of the rack count so nodes distribute evenly.

Step 4: Monitoring (Prometheus + Grafana)

Enable Cassandra metrics by adding telemetry to the K8ssandraCluster spec:

spec:
  cassandra:
    telemetry:
      prometheus:
        enabled: true
  reaper:
    telemetry:
      prometheus:
        enabled: true

This creates ServiceMonitor resources automatically. Install kube-prometheus-stack to scrape them:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install monitoring prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace \
  --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
  --set prometheus.prometheusSpec.serviceMonitorSelector={}

The serviceMonitorSelectorNilUsesHelmValues=false setting is critical — without it, Prometheus ignores the K8ssandra ServiceMonitors because they weren't created by Helm.

Access Grafana:

kubectl port-forward svc/monitoring-grafana 3000:80 -n monitoring
# Default creds: admin / prom-operator

Step 5: Backup and Restore (Medusa)

Medusa runs as a sidecar on each Cassandra pod and streams backups to object storage.

Create a credentials secret

For S3:

apiVersion: v1
kind: Secret
metadata:
  name: medusa-bucket-key
  namespace: k8ssandra-operator
type: Opaque
stringData:
  credentials: |-
    [default]
    aws_access_key_id = YOUR_KEY
    aws_secret_access_key = YOUR_SECRET

For GCS, use a service account JSON key. For Azure, use storage account credentials. See the cloud-specific reference file for details.

Add Medusa to the K8ssandraCluster

spec:
  medusa:
    storageProperties:
      storageProvider: s3              # or google_storage, azure_blobs
      storageSecretRef:
        name: medusa-bucket-key
      bucketName: my-cassandra-backups
      region: us-east-1               # must match actual bucket region
      prefix: production

Trigger a backup

apiVersion: medusa.k8ssandra.io/v1alpha1
kind: MedusaBackupJob
metadata:
  name: backup-2026-03-02
  namespace: k8ssandra-operator
spec:
  cassandraDatacenter: dc1

Restore from backup

apiVersion: medusa.k8ssandra.io/v1alpha1
kind: MedusaRestoreJob
metadata:
  name: restore-2026-03-02
  namespace: k8ssandra-operator
spec:
  cassandraDatacenter: dc1
  backup: backup-2026-03-02

Step 6: Automated Repairs (Reaper)

Reaper is included by default. The autoScheduling config above schedules repairs automatically for all non-system keyspaces. Repairs must run within gc_grace_seconds (default 10 days) to prevent data resurrection — weekly is a safe cadence.

For a centralized Reaper managing multiple clusters, see the cloud-specific reference.

Step 7: TLS Encryption

Generate keystores and truststores

# Generate CA
keytool -genkeypair -alias ca -keyalg RSA -keysize 2048 -validity 3650 \
  -keystore ca-keystore.jks -storepass changeit -dname "CN=CassandraCA"
keytool -exportcert -alias ca -keystore ca-keystore.jks -storepass changeit \
  -file ca-cert.pem -rfc

# Create server keystore
keytool -genkeypair -alias cassandra -keyalg RSA -keysize 2048 -validity 365 \
  -keystore server-keystore.jks -storepass changeit -dname "CN=cassandra"
keytool -certreq -alias cassandra -keystore server-keystore.jks -storepass changeit \
  -file server.csr
keytool -gencert -alias ca -keystore ca-keystore.jks -storepass changeit \
  -infile server.csr -outfile server-cert.pem -rfc
keytool -importcert -alias ca -keystore server-keystore.jks -storepass changeit \
  -file ca-cert.pem -noprompt
keytool -importcert -alias cassandra -keystore server-keystore.jks -storepass changeit \
  -file server-cert.pem

# Create truststore
keytool -importcert -alias ca -keystore server-truststore.jks -storepass changeit \
  -file ca-cert.pem -noprompt

Create Kubernetes secrets

kubectl create secret generic server-encryption-stores \
  --from-file=keystore=server-keystore.jks \
  --from-literal=keystore-password=changeit \
  --from-file=truststore=server-truststore.jks \
  --from-literal=truststore-password=changeit \
  -n k8ssandra-operator

Add encryption to the K8ssandraCluster

spec:
  cassandra:
    config:
      cassandraYaml:
        server_encryption_options:
          internode_encryption: all
          require_client_auth: true
        client_encryption_options:
          enabled: true
          require_client_auth: false   # set true if clients use certs
    serverEncryptionStores:
      keystoreSecretRef:
        name: server-encryption-stores
      truststoreSecretRef:
        name: server-encryption-stores
    clientEncryptionStores:
      keystoreSecretRef:
        name: server-encryption-stores
      truststoreSecretRef:
        name: server-encryption-stores

Step 8: Authentication

The operator auto-generates a superuser secret named <cluster-name>-superuser. Extract the credentials:

kubectl get secret production-superuser -n k8ssandra-operator \
  -o jsonpath='{.data.username}' | base64 -d
kubectl get secret production-superuser -n k8ssandra-operator \
  -o jsonpath='{.data.password}' | base64 -d

To use a custom superuser, create a secret before deploying the cluster:

apiVersion: v1
kind: Secret
metadata:
  name: production-superuser
  namespace: k8ssandra-operator
type: Opaque
stringData:
  username: admin
  password: a-strong-password-here

Step 9: Scaling

Add nodes to an existing datacenter

Increase size in the datacenter spec and re-apply. The operator distributes new nodes across racks evenly.

Add a new datacenter

Add a new entry to datacenters[]. The operator handles token assignment, system keyspace replication, and nodetool rebuild automatically. Annotate the cluster to set replication for user keyspaces in the new DC:

metadata:
  annotations:
    k8ssandra.io/dc-replication: '{"dc2": {"my_keyspace": 3}}'

All user keyspaces should use NetworkTopologyStrategy before adding a DC.

Verification Checklist

After applying the manifest, verify the deployment:

# Check K8ssandraCluster status
kubectl get k8cs -n k8ssandra-operator

# Check all pods are running
kubectl get pods -n k8ssandra-operator

# Check CassandraDatacenter is ready
kubectl get cassandradatacenters -n k8ssandra-operator

# Test CQL connectivity
kubectl exec -it production-dc1-default-sts-0 -n k8ssandra-operator \
  -c cassandra -- cqlsh -u <username> -p <password>

# Verify cluster topology
nodetool status

Common Pitfalls

• Forgot cert-manager: The operator pods will crash. Always install cert-manager first.
• Wrong storage binding mode: Pods hang in Pending. Ensure WaitForFirstConsumer.
• Medusa bucket mismatch: If bucket name or region is wrong, pods won't start.
• Underprovisioned resources: Cassandra needs at least 2Gi heap for production. Set heapSize to roughly 50% of the memory limit, capped at 8G.
• Stargate is deprecated: Do not use Stargate with new deployments; use CQL drivers.
• Rack changes are disruptive: Plan rack topology upfront; changing it later requires careful node replacement.