helm-dev-environment

name

helm-dev-environment

description

Start up, tear down, and configure the local Kubernetes development environment for OpenShell. Uses k3d (Docker-backed k3s) + Skaffold + Helm. Covers cluster lifecycle, optional add-ons (Keycloak OIDC, Envoy Gateway), HA testing, and port mappings. Trigger keywords - local k8s, local cluster, k3d, skaffold, helm dev, start cluster, stop cluster, tear down cluster, delete cluster, create cluster, helm:k3s, helm:skaffold, local dev environment, dev cluster, k8s dev, envoy gateway local, keycloak local, high availability, HA.

Helm Dev Environment

Set up, run, and tear down the local Kubernetes development environment for OpenShell. The stack is: k3d (Docker-backed k3s) for the cluster, Skaffold for image builds and Helm deploys, and the OpenShell Helm chart (deploy/helm/openshell/).

---

Prerequisites

• Docker Desktop (macOS) or Docker Engine (Linux) running
• mise install completed (provides k3d, kubectl, skaffold, helm)

---

Startup

1. Create the cluster

mise run helm:k3s:create

Creates a k3d cluster and merges its kubeconfig into the worktree-local kubeconfig file. Also applies the upstream agent-sandbox CRDs/controller (pinned via AGENT_SANDBOX_VERSION in tasks/scripts/helm-k3s-local.sh, fetched from github.com/kubernetes-sigs/agent-sandbox releases) and preloads the default community sandbox image into k3d so the first sandbox create does not wait on a large registry pull. Traefik is disabled at cluster creation time.

Multi-worktree support: the cluster name is derived from the last component of the current git branch (e.g. branch kube-support/local-dev/tmutch → cluster openshell-dev-tmutch). Each worktree therefore gets its own isolated cluster and its own kubeconfig file. Override with HELM_K3S_CLUSTER_NAME to force a specific name or share one cluster across worktrees.

Port mappings created at cluster time (cannot be changed without recreating):

Host port	Target	Used by
8080	Port 80 via k3d load balancer	Envoy Gateway LoadBalancer service (values-gateway.yaml)

Override with env vars before running helm:k3s:create:

• HELM_K3S_LB_HOST_PORT (default: 8080)
• HELM_K3S_PRELOAD_SANDBOX_IMAGE (default: ghcr.io/nvidia/openshell-community/sandboxes/base:latest; set to an empty value to skip)

2. Deploy OpenShell

Iterative dev (rebuilds on file changes, recommended during active development):

mise run helm:skaffold:dev

One-shot deploy (build once and leave running):

mise run helm:skaffold:run

Supervisor sidecar topology (build once and leave running):

mise run helm:skaffold:run:sidecar

Supervisor sidecar topology with TLS/mTLS enabled (build once and leave running):

mise run helm:skaffold:run:sidecar-mtls

Both commands build the gateway and supervisor images and deploy the OpenShell Helm chart. The sidecar profile renders an openshell-network-init init container for nftables setup and an openshell-supervisor-network runtime sidecar for proxying. Binary-aware policy mode runs that sidecar as UID 0 with SYS_PTRACE and DAC_READ_SEARCH; relaxed mode can run it as the configured proxy UID. The sidecar-mTLS profile reuses ci/values-sidecar.yaml and restores server.disableTls=false inline for Skaffold. The pkiInitJob hook (a pre-install Job that runs openshell-gateway generate-certs) generates mTLS secrets on first install. Envoy Gateway opt-in; see the Optional Add-ons section below.

The gateway Service uses ClusterIP. Access is via Envoy Gateway (port 8080) or kubectl port-forward.

HA test deploy (two gateway replicas + external PostgreSQL Secret): uncomment #- ci/values-high-availability.yaml in deploy/helm/openshell/skaffold.yaml, create the Secret named openshell-ha-pg with a uri key, then run mise run helm:skaffold:run or mise run helm:skaffold:dev.

TLS behaviour

ci/values-skaffold.yaml sets server.disableTls: true, so Skaffold-based deploys run plaintext by default. To test sidecar topology with TLS enabled, use mise run helm:skaffold:run:sidecar-mtls.

Mode	server.disableTls	Gateway scheme
Skaffold dev (default)	true	http://
TLS enabled	false (or omitted)	https://

Connecting via port-forward

Port 8080 is already bound by the k3d load balancer when Envoy Gateway is active, so the port-forward uses local port 8090 to avoid a collision:

KUBECONFIG=kubeconfig kubectl port-forward -n openshell svc/openshell 8090:8080

Plaintext (default Skaffold deploy):

openshell sandbox list --gateway-endpoint http://localhost:8090

With mTLS enabled — extract the client cert the PKI hook wrote to the cluster, then place it where the CLI expects it. Run once after each fresh install:

mkdir -p ~/.config/openshell/gateways/openshell/mtls
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.ca\.crt}'  | base64 -d > ~/.config/openshell/gateways/openshell/mtls/ca.crt
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.crt
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.tls\.key}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.key

The server cert SANs include localhost and 127.0.0.1, so hostname verification passes over a port-forward without any extra flags:

openshell sandbox list --gateway-endpoint https://localhost:8090

---

Teardown

Remove the Helm releases (keep cluster)

mise run helm:skaffold:delete

For a sidecar-profile deployment:

mise run helm:skaffold:delete:sidecar

Delete the cluster entirely

mise run helm:k3s:delete

This removes the k3d cluster and all resources. Kubeconfig context is left behind but will point to a deleted cluster — safe to ignore or clean up manually.

---

Optional Add-ons

Each add-on requires uncommenting the corresponding valuesFiles entry in deploy/helm/openshell/skaffold.yaml before running helm:skaffold:dev or helm:skaffold:run.

Envoy Gateway (Gateway API / GRPCRoute)

Envoy Gateway is already installed by Skaffold (the envoy-gateway Helm release in skaffold.yaml). To activate routing:

1. Uncomment #- values-gateway.yaml in skaffold.yaml
2. Redeploy: mise run helm:skaffold:run
3. Apply the GatewayClass: mise run helm:gateway:apply
4. Access: http://127.0.0.1:8080

values-gateway.yaml creates a Gateway (listener on port 80, class eg) and a GRPCRoute in the openshell namespace. Envoy Gateway provisions a LoadBalancer service for the proxy; klipper-lb binds it to hostPort 80, reachable via the 8080:80 load balancer port mapping.

Keycloak OIDC

One-time setup — only needed once per cluster lifetime:

mise run keycloak:k8s:setup

This deploys Keycloak (quay.io/keycloak/keycloak:24.0) into the keycloak namespace, imports the openshell realm from scripts/keycloak-realm.json, and prints a port-forward command for acquiring tokens from the CLI.

Then activate OIDC in the OpenShell Helm chart:

1. Uncomment #- ci/values-keycloak.yaml in skaffold.yaml
2. Redeploy: mise run helm:skaffold:run

To remove Keycloak:

mise run keycloak:k8s:teardown

SPIRE / SPIFFE Provider Token Grants

Skaffold can install SPIRE with the SPIFFE hardened Helm charts. To activate SPIFFE JWT-SVIDs for dynamic provider token grants:

1. Uncomment the spire-crds and spire releases in deploy/helm/openshell/skaffold.yaml
2. Uncomment #- ci/values-spire.yaml in the OpenShell release values files
3. Redeploy: mise run helm:skaffold:run

ci/values-spire-stack.yaml configures the local SPIRE trust domain as openshell.local and adds a ClusterSPIFFEID that maps sandbox pod annotations to spiffe://openshell.local/openshell/sandbox/<sandbox-id>. OpenShell mounts the SPIFFE CSI Workload API socket at /spiffe-workload-api/spire-agent.sock into sandbox pods for provider token grants. Supervisor-to-gateway authentication remains on the Kubernetes ServiceAccount bootstrap and gateway-minted sandbox JWT path.

---

Cluster Lifecycle (stop/start)

Stop the cluster without losing state (faster than delete/recreate):

mise run helm:k3s:stop
mise run helm:k3s:start

Check cluster status:

mise run helm:k3s:status

---

Helm Chart Checks

Run the chart lint task before changing Helm templates, values overlays, or Skaffold inputs:

mise run helm:lint

If Helm reports missing chart dependencies, remove the specific stale subchart archive or directory named by the error from deploy/helm/openshell/charts/, then rerun the lint task.

For example, when lint reports chart metadata is missing these dependencies: postgresql, remove stale PostgreSQL chart artifacts:

rm -f deploy/helm/openshell/charts/postgresql-*.tgz
rm -rf deploy/helm/openshell/charts/postgresql
mise run helm:lint

The charts/ directory is ignored and regenerated by helm dependency build for dependencies still declared in Chart.yaml.

---

Key Files

Path	Purpose
deploy/helm/openshell/skaffold.yaml	Skaffold config — images, Helm releases, values overlays
deploy/helm/openshell/values.yaml	Default Helm values
deploy/helm/openshell/ci/values-skaffold.yaml	Dev overrides (image pull policy, TLS disabled for local Skaffold)
deploy/helm/openshell/ci/values-cert-manager.yaml	cert-manager PKI overlay (opt-in; disables pkiInitJob)
deploy/helm/openshell/ci/values-gateway.yaml	Envoy Gateway GRPCRoute + Gateway overlay
deploy/helm/openshell/ci/values-high-availability.yaml	HA test overlay (replicaCount: 2 with external PostgreSQL Secret)
deploy/helm/openshell/ci/values-keycloak.yaml	Keycloak OIDC overlay
deploy/helm/openshell/ci/values-sidecar.yaml	Supervisor sidecar topology overlay for Kubernetes e2e/dev
deploy/helm/openshell/ci/values-spire.yaml	SPIFFE/SPIRE provider token grant overlay
deploy/helm/openshell/ci/values-spire-stack.yaml	SPIRE hardened chart values for local dev
deploy/helm/openshell/ci/values-tls-disabled.yaml	Lint-only: TLS + auth disabled (reverse-proxy edge termination)
deploy/kube/manifests/envoy-gateway-openshell.yaml	GatewayClass for Envoy Gateway (mise run helm:gateway:apply)
tasks/scripts/helm-k3s-local.sh	k3d cluster create/delete/start/stop/status
tasks/scripts/keycloak-k8s-setup.sh	Keycloak deploy + realm import