helm-dev-environment

# Helm Dev Environment Set up, run, and tear down the local Kubernetes development environment for OpenShell. The stack is: **k3d** (Docker-backed k3s) for the cluster, **Skaffold** for image builds and Helm deploys, and the **OpenShell Helm chart** (`deploy/helm/openshell/`). --- ## Prerequisites - Docker Desktop (macOS) or Docker Engine (Linux) running - `mise install` completed (provides `k3d`, `kubectl`, `skaffold`, `helm`) --- ## Startup ### 1. Create the cluster ```bash mise run helm:k3s:create ``` Creates a k3d cluster and merges its kubeconfig into the worktree-local `kubeconfig` file. When the named cluster already exists, the task starts any stopped containers and refreshes same-named kubeconfig entries so a recreated load balancer's current API port takes effect. Also applies the upstream agent-sandbox CRDs/controller (pinned via `AGENT_SANDBOX_VERSION` in `tasks/scripts/helm-k3s-local.sh`, fetched from `github.com/kubernetes-sigs/agent-sandbox` releases), enables its OTLP tracing on v0.5 and later, installs an OTLP trace collector and UI in the `observability` namespace, and preloads the default sandbox image into k3d so the first sandbox create does not wait on a large registry pull. Traefik is disabled at cluster creation time. **Multi-worktree support:** the cluster name is derived from the last component of the current git branch (e.g. branch `chore/1234-local-dev/octocat` → cluster `openshell-dev-octocat`). Each worktree therefore gets its own isolated cluster and its own `kubeconfig` file. Override with `HELM_K3S_CLUSTER_NAME` to force a specific name or share one cluster across worktrees. Port mappings created at cluster time (cannot be changed without recreating): | Host port | Target | Used by | |-----------|--------|---------| | `8080` | Port `80` via k3d load balancer | Envoy Gateway LoadBalancer service (`values-gateway.yaml`) | Override with env vars before running `helm:k3s:create`: - `HELM_K3S_LB_HOST_PORT` (default: `8080`) - `HELM_K3S_PRELOAD_SANDBOX_IMAGE` (default: `nvcr.io/nvidia/base/ubuntu:24.04`; set to an empty value to skip) - `HELM_K3S_COLLECTOR_IMAGE` (default: `mcr.microsoft.com/dotnet/aspire-dashboard:latest`) - `HELM_K3S_COLLECTOR_HEALTH_TIMEOUT` (default: `120` seconds) ### Shared supervisor plumbing Set `gatewayConfig.openshell.drivers.kubernetes.supervisor_topology` using a values file (the table key is literally `openshell.drivers.kubernetes`): ```yaml gatewayConfig: openshell.drivers.kubernetes: supervisor_topology: gateway ``` The gateway chart creates a shared supervisor Deployment (`supervisor.replicas`, default one), Service, NetworkPolicy, and an empty `<release>-supervisor-token` Secret in the release namespace, regardless of shared/managed/operator workspace mode. The gateway mints a supervisor token from its sandbox-JWT signing key, writes it into that Secret (the gateway Role may `get`/`patch` only that Secret), and renews it at half its lifetime; the supervisor mounts the Secret and presents the token on the private `openshell.supervisor.host.v1` assignment stream, which accepts nothing else. The topology requires gateway TLS and the signing key (`server.disableTls` is rejected; the gateway refuses to start without `gateway_jwt`). New sandbox generations run as sandbox instances in that process. The chart starts `--mode=shared`, sets `--gateway-endpoint` to the gateway gRPC URL, mounts the token at `--token-file`, and binds `--address` (5501) to a plain TCP health listener for the `tcpSocket` probes; nothing dials the supervisor but sandbox runtimes (5500) and the kubelet. Build the gateway, supervisor, and sandbox images from the same branch: all three must support the routed runtime connection and the assignment stream. `workspace` is reserved and rejected. Omit the field to disable it. `mise run e2e:kubernetes:shared-supervisor` runs the focused e2e (`ci/values-supervisor.yaml`). The task prepends that overlay to any colon-separated `OPENSHELL_E2E_KUBE_EXTRA_VALUES` list, so layer a workspace-mode overlay to cover managed namespaces: ```bash OPENSHELL_E2E_KUBE_EXTRA_VALUES=deploy/helm/openshell/ci/values-workspace-managed.yaml \ OPENSHELL_E2E_KUBE_IMAGE_PULL_SECRET=e2e-regcred \ mise run e2e:kubernetes:shared-supervisor ``` ### Shared supervisor high availability The shared supervisor is always a pool of `supervisor.replicas` members (default 1), or an HPA-managed pool when `supervisor.autoscaling.enabled=true`. `ci/values-supervisor-autoscaling.yaml` enables the memory HPA (default `512Mi` average per Pod, two to four members, one removal every two minutes after five minutes of stabilization). It requires the resource metrics API, usually Metrics Server. The HPA omits the Deployment's fixed replica count and ignores `supervisor.replicas`; top-level `autoscaling` still controls only the gateway. Do not combine the HPA overlay with the manual 1→2→1 scaling E2E: that test expects Helm to own replicas. Changing HPA bounds or target must leave the gateway config and supervisor Pod template unchanged, as checked by `deploy/helm/test-supervisor-pool-scaling.sh`. With the gateway topology the chart renders the gateway's `[openshell.gateway.supervisor_pool]` tuning table (only `recovery_timeout_secs` from `supervisor.recoveryTimeoutSeconds`, default 60; the driver's `shared_supervisor` table stays unchanged), a RollingUpdate Deployment (`maxSurge: 1`, `maxUnavailable: 0`) with a 60-second grace period, no affinity unless `supervisor.affinity` is set (top-level `affinity` applies only to the gateway), the pool identity env vars (`OPENSHELL_SUPERVISOR_POOL_MEMBER_ID` and `_ADVERTISE_IP` from the Pod name and IP), and a `maxUnavailable: 1` PodDisruptionBudget unless `supervisor.podDisruptionBudget.enabled=false` (default `true`; toggling it changes only the PodDisruptionBudget). Each supervisor resource has its own template: `supervisor.yaml` (Deployment), `supervisor-service.yaml`, `supervisor-networkpolicy.yaml`, `supervisor-token-secret.yaml`, `supervisor-pdb.yaml`, and `supervisor-hpa.yaml`. `ci/values-supervisor-ha.yaml` selects two replicas and a 60-second recovery timeout. The topology needs gateway TLS: the e2e harness keeps it on with `ci/values-tls-port-forward.yaml` and `OPENSHELL_E2E_KUBE_TLS=1`, which reaches the gateway over mTLS through its port-forward. `mise run e2e:kubernetes:shared-supervisor-ha` sets both, creates a task-scoped k3d cluster, kills the supervisor Pod that owns a sandbox, and checks that the sandbox keeps its Pod, generation, main process, and in-memory state. The rendered gateway configuration and supervisor Pod template do not depend on `supervisor.replicas`; `deploy/helm/test-supervisor-pool-scaling.sh` (part of `mise run helm:test`) renders a pool at 1, 2, and 3 replicas and fails if anything but the replica count differs. `mise run e2e:kubernetes:shared-supervisor-scaling` installs a one-member pool (`ci/values-supervisor-ha-scaling.yaml`), then the test runs `helm upgrade --reuse-values --set supervisor.replicas=2` and back to 1 against the chart the harness installed (`OPENSHELL_E2E_KUBE_CHART`), probing exec and policy-mediated networking throughout. It checks that scaling up moves nothing, that the member removed by scaling down hands its sandboxes to the other (`cause=leaving`), that no workload, gateway Pod, or remaining member restarts, and that a new sandbox runs after each step. It upgrades its release, so it runs on a release of its own; the two pool tests also share a nextest test group that never runs them at once. Against an existing cluster whose kubeconfig the shell environment may override, set `OPENSHELL_E2E_KUBECONFIG` (passed as `--kubeconfig` to kubectl and helm). On k3d, deploy the pool with the Skaffold profile, which layers the overlay and turns gateway TLS back on over `ci/values-skaffold.yaml`: ```bash cd deploy/helm/openshell KUBECONFIG=../../../kubeconfig skaffold run -p shared-supervisor-high-availability cd ../../.. ``` The plaintext registration from `helm:skaffold:run` does not apply. Keep `mise run helm:k3s:forward` running and set up the client certificate as in "With mTLS enabled" under [Connecting through the forwarding task](#connecting-through-the-forwarding-task). Both members share the single k3d node, so exercise Pod failures: ```bash export KUBECONFIG=kubeconfig OPENSHELL_GATEWAY_ENDPOINT=https://localhost:8090 openshell sandbox create --detach --name ha-probe -- sh -c 'echo $$ > /sandbox/main.pid; exec sleep infinity' openshell sandbox exec --name ha-probe --no-tty -- cat /sandbox/main.pid openshell sandbox get ha-probe # note the sandbox ID # Supervisor logs name the instance by its route, not its sandbox ID. kubectl get sandboxes.agents.x-k8s.io -A -l openshell.ai/sandbox-name=ha-probe \ -o jsonpath='{.items[0].metadata.annotations.openshell\.ai/sandbox-runtime-supervisor-session}' # The member whose latest admission, removal, or release line names the route owns it. kubectl -n openshell logs -l app.kubernetes.io/name=openshell-supervisor --prefix --tail=-1 \ | grep -E '(admitted assigned|removing withdrawn|relinquishing a) sandbox instance.* <route>$' # Force-delete the owner, the closest local stand-in for a crash. kubectl -n openshell delete pod <owner-pod> --grace-period=0 --force kubectl -n openshell logs -l app.kubernetes.io/name=openshell,app.kubernetes.io/instance=openshell \ -c openshell-gateway --prefix --tail=200 | grep 'taking over a shared sandbox instance' kubectl -n openshell logs -l openshell.ai/sandbox-id=<sandbox-id>,openshell.ai/boundary-role=workload \ --tail=200 | grep -E 'sandbox-supervisor-|Supervisor takeover confirmed' openshell sandbox exec --name ha-probe --no-tty -- cat /sandbox/main.pid # same PID ``` A plain `kubectl delete pod` exercises the graceful leave instead: the member announces it is leaving and the gateway hands its instances over at once. A member that stops without announcing it, as on node loss, is declared dead only after the 15-second membership TTL. `kubectl -n openshell rollout restart deployment/openshell-supervisor` replaces every member one at a time. The PID must survive each case, while an open `sandbox connect` or `exec` stream breaks and must reconnect. A `sandbox-supervisor-recovery-expired` finding in the workload log means no replacement confirmed within the recovery timeout, so the runtime terminated the workload. For RFC 0017 app-listener validation, enable the listener and supply the app TLS Secret on the shared Deployment, plus a separate NetworkPolicy allowing the test client/proxy to reach its app port. Runtime-port access does not grant app access. Discovery returns the acknowledged owner's Pod IP and app port; no headless app Service is needed. Exercise HTTP/WebSocket reconnection through graceful downscale and abrupt loss, preserving workload generation/PID/in-memory state and the public hostname. Verify stale-owner rejection independently of the replacement's success. Shared serving leases expire within fifteen seconds of issuance and do not renew from configuration alone. Do not claim seamless socket migration or infer external ingress readiness from an advertisement. ### 2. Deploy OpenShell **Iterative dev** (rebuilds on file changes, recommended during active development): ```bash mise run helm:skaffold:dev ``` **One-shot deploy** (build once and leave running): ```bash mise run helm:skaffold:run ``` Resource admission defaults to enabled and caller driver config to disabled. Driver-config scenarios need an explicit `allowDriverConfig` opt-in; external attachments also need administrator-controlled approval labels in the target namespace. GPU attachments and operator-selected image-pull Secrets are exempt from labels. Managed workspace image-pull Secrets are copied from the configured source in the gateway namespace; do not grant approval to the gateway database PVC or disable admission to make tests pass. The Skaffold flow builds distinct `gateway`, `sandbox`, and `supervisor` images and deploys the OpenShell Helm chart. The Kubernetes driver creates a capability-free workload Pod and a directly managed capability-free supervisor Pod. One namespace-wide NetworkPolicy denies direct egress from every OpenShell workload Pod. New sandbox generations use `runtime-initiated` control: the sandbox runtime dials its supervisor through `os-supervisor-<id>`. The legacy `os-boundary-<id>` Service keeps selecting the workload for old gateways during rolling upgrades. New gateways create both Services and bind the control-mode annotation to its runtime generation. The driver adds a per-sandbox `os-control-<id>` NetworkPolicy for that path only. Existing running generations keep their connection until replaced; stop/start and gateway recovery that recreates both Pods upgrade legacy generations automatically. Use compatible gateway, supervisor, and sandbox runtime images together; there is no legacy rollout flag. The `pkiInitJob` hook (a pre-install Job that runs `openshell-gateway generate-certs`) generates gateway and CLI TLS secrets on first install. Supervisor Pods project only `ca.crt` and authenticate gateway RPCs with sandbox bearer tokens. User client certificates and private keys remain outside supervisor and workload Pods. The default Skaffold values export gateway and Kubernetes-driver traces to the collector service installed by `helm:k3s:create`. Envoy Gateway is opt-in; see the Optional Add-ons section. The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or the unified local forwarding task: ```bash mise run helm:k3s:forward ``` The task forwards OTLP/gRPC to `http://127.0.0.1:4317` and the trace UI to `http://127.0.0.1:18888`. When Skaffold has deployed a Kubernetes gateway, it also forwards the gateway to `http://127.0.0.1:8090`; otherwise it continues with the collector ports only. A successful plaintext `helm:skaffold:run` registers the gateway under the worktree-specific k3d cluster name and selects it as the active gateway. Keep the forwarding task running while using those endpoints. ### Viewing local traces The gateway exports OTLP/gRPC to `http://openshell-collector.observability.svc.cluster.local:4317` through the default Skaffold values. Forward OTLP/gRPC and the trace UI to the host: ```bash mise run helm:k3s:forward ``` Open `http://127.0.0.1:18888` and exercise the gateway to inspect gateway and Kubernetes compute-driver spans under their distinct service names, along with Agent Sandbox controller reconciliation spans linked through the Sandbox trace-context annotation. The same command exposes OTLP/gRPC on `http://127.0.0.1:4317` and, when deployed, the Kubernetes gateway on `http://127.0.0.1:8090`. The local `gateway:docker`, `gateway:podman`, and `gateway:vm` tasks detect the collector listener at startup and enable trace export, OCSF log export, and the agent trace relay only while it is reachable. Spans an agent exports from inside a sandbox arrive through the supervisor's relay with `openshell.telemetry.source=agent` and share `openshell.sandbox.id` with that sandbox's supervisor spans. The Skaffold profile for HA reverse-proxy development is available from `deploy/helm/openshell/`: ```bash # Two gateway replicas + external PostgreSQL Secret + Envoy Gateway + Gateway API route. KUBECONFIG=../../../kubeconfig skaffold run -p high-availability ``` The `high-availability` profile expects a Secret named `openshell-ha-pg` in the `openshell` namespace with a `uri` key. For local manual testing, either create your own PostgreSQL Secret or use the e2e PostgreSQL fixture manifest in `e2e/kubernetes/postgres-fixture.yaml`. For the `high-availability` profile, return to the repository root and apply the GatewayClass and BackendTrafficPolicy manifest after Skaffold has installed Envoy Gateway: ```bash KUBECONFIG=kubeconfig mise run helm:gateway:apply ``` The BackendTrafficPolicy disables Envoy request and stream-duration timeouts for OpenShell's `GRPCRoute`. Keep that policy in `deploy/kube/manifests/envoy-gateway-openshell.yaml`, not in the Helm chart; it is required for long-lived gRPC create/watch/exec/relay streams during gateway rollouts and scale events. ### TLS behaviour `ci/values-skaffold.yaml` sets `server.disableTls: true`, so Skaffold-based deploys run plaintext by default. Override `server.disableTls=false` to exercise TLS/mTLS. | Mode | `server.disableTls` | Gateway scheme | |------|---------------------|----------------| | Skaffold dev (default) | `true` | `http://` | | TLS enabled | `false` (or omitted) | `https://` | ### Connecting through the forwarding task Port `8080` is already bound by the k3d load balancer when Envoy Gateway is active, so the forwarding task uses local port `8090` for the gateway. In a second terminal, confirm that the gateway is registered and active: ```bash openshell gateway list ``` **Plaintext (default Skaffold deploy):** ```bash openshell sandbox list ``` **With mTLS enabled** — extract the client cert the PKI hook wrote to the cluster, then place it where the CLI expects it. Run once after each fresh install: ```bash mkdir -p ~/.config/openshell/gateways/openshell/mtls KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \ -o jsonpath='{.data.ca\.crt}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/ca.crt KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \ -o jsonpath='{.data.tls\.crt}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.crt KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \ -o jsonpath='{.data.tls\.key}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.key ``` The server cert SANs include `localhost` and `127.0.0.1`, so hostname verification passes over a port-forward without any extra flags: ```bash openshell sandbox list --gateway-endpoint https://localhost:8090 ``` --- ## Teardown ### Remove the Helm releases (keep cluster) ```bash mise run helm:skaffold:delete ``` ### Delete the cluster entirely ```bash mise run helm:k3s:delete ``` This removes the k3d cluster and all resources. Kubeconfig context is left behind