debug-openshell-cluster

name

debug-openshell-cluster

description

Debug why a openshell cluster failed to start or is unhealthy. Use when the user has a failed `openshell gateway start`, cluster health check failure, or wants to diagnose cluster infrastructure issues. Trigger keywords - debug cluster, cluster failing, cluster not starting, deploy failed, cluster troubleshoot, cluster health, cluster diagnose, why won't my cluster start, health check failed, gateway start failed, gateway not starting.

Debug OpenShell Cluster

Diagnose why a openshell cluster failed to start after openshell gateway start.

Use only openshell CLI commands (openshell status, openshell doctor logs, openshell doctor exec) to inspect and fix the cluster. Do not use raw docker, ssh, or kubectl commands directly — always go through the openshell doctor interface. The CLI auto-resolves local vs remote gateways, so the same commands work everywhere.

Overview

openshell gateway start creates a Docker container running k3s with the OpenShell server deployed via Helm. The deployment stages, in order, are:

1. Pre-deploy check: openshell gateway start in interactive mode prompts to reuse (keep volume, clean stale nodes) or recreate (destroy everything, fresh start). mise run cluster always recreates before deploy.
2. Ensure cluster image is available (local build or remote pull)
3. Create Docker network (openshell-cluster) and volume (openshell-cluster-{name})
4. Create and start a privileged Docker container (openshell-cluster-{name})
5. Wait for k3s to generate kubeconfig (up to 60s)
6. Clean stale nodes: Remove any NotReady k3s nodes left over from previous container instances that reused the same persistent volume
7. Prepare local images (if OPENSHELL_PUSH_IMAGES is set): In internal registry mode, bootstrap waits for the in-cluster registry and pushes tagged images there. In external mode, bootstrap uses legacy ctr -n k8s.io images import push-mode behavior.
8. Reconcile TLS PKI: Load existing TLS secrets from the cluster; if missing, incomplete, or malformed, generate fresh PKI (CA + server + client certs). Apply secrets to cluster. If rotation happened and the OpenShell workload is already running, rollout restart and wait for completion (failed rollout aborts deploy).
9. Store CLI mTLS credentials: Persist client cert/key/CA locally for CLI authentication.
10. Wait for cluster health checks to pass (up to 6 min):
    • k3s API server readiness (/readyz)
    • openshell statefulset ready in openshell namespace
    • TLS secrets openshell-server-tls and openshell-client-tls exist in openshell namespace
    • Sandbox supervisor binary exists at /opt/openshell/bin/openshell-sandbox (emits HEALTHCHECK_MISSING_SUPERVISOR marker if absent)

For local deploys, metadata endpoint selection now depends on Docker connectivity:

• default local Docker socket (unix:///var/run/docker.sock): https://127.0.0.1:{port} (default port 8080)
• TCP Docker daemon (DOCKER_HOST=tcp://<host>:<port>): https://<host>:{port} for non-loopback hosts

The host port is configurable via --port on openshell gateway start (default 8080) and is stored in ClusterMetadata.gateway_port.

The TCP host is also added as an extra gateway TLS SAN so mTLS hostname validation succeeds.

The default cluster name is openshell. The container is openshell-cluster-{name}.

Prerequisites

• Docker must be running (locally or on the remote host)
• The openshell CLI must be available
• For remote clusters: SSH access to the remote host

Tools Available

All diagnostics go through three openshell commands. They auto-resolve local vs remote gateways — the same commands work for both:

# Quick connectivity check
openshell status

# Fetch container logs
openshell doctor logs --lines 100
openshell doctor logs --tail          # stream live

# Run any command inside the gateway container (KUBECONFIG is pre-configured)
openshell doctor exec -- kubectl get pods -A
openshell doctor exec -- kubectl -n openshell logs statefulset/openshell --tail=100
openshell doctor exec -- cat /etc/rancher/k3s/registries.yaml
openshell doctor exec -- df -h /
openshell doctor exec -- free -h
openshell doctor exec -- sh           # interactive shell

Workflow

When the user asks to debug a cluster failure, run diagnostics automatically through the steps below in order. Stop and report findings as soon as a root cause is identified. Do not ask the user to choose which checks to run.

Determine Context

Before running commands, establish:

1. Cluster name: Default is openshell, giving container name openshell-cluster-openshell
2. Remote or local: The openshell doctor commands auto-resolve this from gateway metadata — no special flags needed for the active gateway
3. Config directory: ~/.config/openshell/gateways/{name}/

Step 0: Quick Connectivity Check

Run openshell status first. This immediately reveals:

• Which gateway and endpoint the CLI is targeting
• Whether the CLI can reach the server (mTLS handshake success/failure)
• The server version if connected

Common errors at this stage:

• tls handshake eof: The server isn't running or mTLS credentials are missing/mismatched
• connection refused: The container isn't running or port mapping is broken
• No gateway configured: No gateway has been deployed yet

Step 1: Check Container Logs

Get recent container logs to identify startup failures:

openshell doctor logs --lines 100

Look for:

• DNS resolution failures in the entrypoint script
• k3s startup errors (certificate issues, port binding failures)
• Manifest copy errors from /opt/openshell/manifests/
• iptables or cgroup errors (privilege/capability issues)

Step 2: Check k3s Cluster Health

Verify k3s itself is functional:

# API server readiness
openshell doctor exec -- kubectl get --raw="/readyz"

# Node status
openshell doctor exec -- kubectl get nodes -o wide

# All pods
openshell doctor exec -- kubectl get pods -A -o wide

If /readyz fails, k3s is still starting or has crashed. Check container logs (Step 1).

If pods are in CrashLoopBackOff, ImagePullBackOff, or Pending, investigate those pods specifically.

Also check for node pressure conditions that cause the kubelet to evict pods and reject scheduling:

# Check node conditions (DiskPressure, MemoryPressure, PIDPressure)
openshell doctor exec -- kubectl get nodes -o jsonpath="{range .items[*]}{.metadata.name}{range .status.conditions[*]} {.type}={.status}{end}{\"\n\"}{end}"

# Check disk usage inside the container
openshell doctor exec -- df -h /

# Check memory usage
openshell doctor exec -- free -h

If any pressure condition is True, pods will be evicted and new ones rejected. The bootstrap now detects HEALTHCHECK_NODE_PRESSURE markers from the health-check script and aborts early with a clear diagnosis. To fix: free disk/memory on the host, then recreate the gateway.

Step 3: Check OpenShell Server StatefulSet

The OpenShell server is deployed via a HelmChart CR as a StatefulSet named openshell in the openshell namespace. Check its status:

# StatefulSet status
openshell doctor exec -- kubectl -n openshell get statefulset/openshell -o wide

# OpenShell pod logs
openshell doctor exec -- kubectl -n openshell logs statefulset/openshell --tail=100

# Describe statefulset for events
openshell doctor exec -- kubectl -n openshell describe statefulset/openshell

# Helm install job logs (the job that installs the OpenShell chart)
openshell doctor exec -- kubectl -n kube-system logs -l job-name=helm-install-openshell --tail=200

Common issues:

• Replicas 0/0: The StatefulSet has been scaled to zero — no pods are running. This can happen after a failed deploy, manual scale-down, or Helm values misconfiguration. Fix: openshell doctor exec -- kubectl -n openshell scale statefulset openshell --replicas=1
• ImagePullBackOff: The component image failed to pull. In internal mode, verify internal registry readiness and pushed image tags (Step 5). In external mode, check /etc/rancher/k3s/registries.yaml credentials/endpoints and DNS (Step 8). Default external registry is ghcr.io/nvidia/openshell/ (public, no auth required). If using a private registry, ensure --registry-username and --registry-token (or OPENSHELL_REGISTRY_USERNAME/OPENSHELL_REGISTRY_TOKEN) were provided during deploy.
• CrashLoopBackOff: The server is crashing. Check pod logs for the actual error.
• Pending: Insufficient resources or scheduling constraints.

Step 4: Check Networking

The OpenShell server is exposed via a NodePort service on port 30051:

# Service status
openshell doctor exec -- kubectl -n openshell get service/openshell

Expected port: 30051/tcp (mapped to configurable host port, default 8080; set via --port on deploy).

Step 5: Check Image Availability

Component images (server, sandbox) can reach kubelet via two paths:

Local/external pull mode (default local via mise run cluster): Local images are tagged to the configured local registry base (default 127.0.0.1:5000/openshell/*), pushed to that registry, and pulled by k3s via registries.yaml mirror endpoint (typically host.docker.internal:5000). The cluster task pushes prebuilt local tags (openshell/*:dev, falling back to localhost:5000/openshell/*:dev or 127.0.0.1:5000/openshell/*:dev).

# Verify image refs currently used by openshell deployment
openshell doctor exec -- kubectl -n openshell get statefulset openshell -o jsonpath="{.spec.template.spec.containers[*].image}"

# Verify registry mirror/auth endpoint configuration
openshell doctor exec -- cat /etc/rancher/k3s/registries.yaml

Legacy push mode: Images are imported into the k3s containerd k8s.io namespace.

# Check if images were imported into containerd (k3s default namespace is k8s.io)
openshell doctor exec -- ctr -a /run/k3s/containerd/containerd.sock images ls | grep openshell

External pull mode (remote deploy, or local with OPENSHELL_REGISTRY_HOST/IMAGE_REPO_BASE pointing at a non-local registry): Images are pulled from an external registry at runtime. The entrypoint generates /etc/rancher/k3s/registries.yaml.

# Verify registries.yaml exists and has credentials
openshell doctor exec -- cat /etc/rancher/k3s/registries.yaml

# Test pulling an image manually from inside the cluster
openshell doctor exec -- crictl pull ghcr.io/nvidia/openshell/gateway:latest

If registries.yaml is missing or has wrong values, verify env wiring (OPENSHELL_REGISTRY_HOST, OPENSHELL_REGISTRY_INSECURE, username/password for authenticated registries).

Step 6: Check mTLS / PKI

TLS certificates are generated by the openshell-bootstrap crate (using rcgen) and stored as K8s secrets before the Helm release installs. There is no PKI job or cert-manager — certificates are applied directly via kubectl apply.

# Check if the three TLS secrets exist
openshell doctor exec -- kubectl -n openshell get secret openshell-server-tls openshell-server-client-ca openshell-client-tls

# Inspect server cert expiry (if openssl is available in the container)
openshell doctor exec -- sh -c 'kubectl -n openshell get secret openshell-server-tls -o jsonpath="{.data.tls\.crt}" | base64 -d | openssl x509 -noout -dates 2>/dev/null || echo "openssl not available"'

# Check if CLI-side mTLS files exist locally
ls -la ~/.config/openshell/gateways/<name>/mtls/

On redeploy, bootstrap reuses existing secrets if they are valid PEM. If secrets are missing or malformed, fresh PKI is generated and the OpenShell workload is automatically restarted. If the rollout restart fails after rotation, the deploy aborts and CLI-side certs are not updated. Certificates use rcgen defaults (effectively never expire).

If the local mTLS files are missing but the secrets exist in the cluster, you can extract them manually:

mkdir -p ~/.config/openshell/gateways/<name>/mtls
openshell doctor exec -- kubectl -n openshell get secret openshell-client-tls -o jsonpath='{.data.ca\.crt}' | base64 -d > ~/.config/openshell/gateways/<name>/mtls/ca.crt
openshell doctor exec -- kubectl -n openshell get secret openshell-client-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > ~/.config/openshell/gateways/<name>/mtls/tls.crt
openshell doctor exec -- kubectl -n openshell get secret openshell-client-tls -o jsonpath='{.data.tls\.key}' | base64 -d > ~/.config/openshell/gateways/<name>/mtls/tls.key

Common mTLS issues:

• Secrets missing: The openshell namespace may not have been created yet (Helm controller race). Bootstrap waits up to 2 minutes for the namespace.
• mTLS mismatch after manual secret deletion: Delete all three secrets and redeploy — bootstrap will regenerate and restart the workload.
• CLI can't connect after redeploy: Check that ~/.config/openshell/gateways/<name>/mtls/ contains ca.crt, tls.crt, tls.key and that they were updated at deploy time.
• Local mTLS files missing: The gateway was deployed but CLI credentials weren't persisted (e.g., interrupted deploy). Extract from the cluster secret as shown above.

Step 7: Check Kubernetes Events

Events catch scheduling failures, image pull errors, and resource issues:

openshell doctor exec -- kubectl get events -A --sort-by=.lastTimestamp | tail -n 50

Look for:

• FailedScheduling — resource constraints
• ImagePullBackOff / ErrImagePull — registry auth failure or DNS issue (check /etc/rancher/k3s/registries.yaml)
• CrashLoopBackOff — application crashes
• OOMKilled — memory limits too low
• FailedMount — volume issues

Step 8: Check DNS Resolution

DNS misconfiguration is a common root cause, especially on remote/Linux hosts:

# Check the resolv.conf k3s is using
openshell doctor exec -- cat /etc/rancher/k3s/resolv.conf

# Test DNS resolution from inside the container
openshell doctor exec -- sh -c 'nslookup google.com || wget -q -O /dev/null http://google.com && echo "network ok" || echo "network unreachable"'

Check the entrypoint's DNS decision in the container logs:

openshell doctor logs --lines 20

The entrypoint script selects DNS resolvers in this priority:

1. Viable nameservers from /etc/resolv.conf (not loopback/link-local)
2. Docker ExtServers from /etc/resolv.conf comments
3. Host gateway IP (Docker Desktop only, 192.168.*)
4. Fallback to 8.8.8.8 / 8.8.4.4

If DNS is broken, all image pulls from the distribution registry will fail, as will pods that need external network access.

Common Failure Patterns

Symptom	Likely Cause	Fix
tls handshake eof from openshell status	Server not running or mTLS credentials missing/mismatched	Check StatefulSet replicas (Step 3) and mTLS files (Step 6)
StatefulSet 0/0 replicas	StatefulSet scaled to zero (failed deploy, manual scale-down, or Helm misconfiguration)	openshell doctor exec -- kubectl -n openshell scale statefulset openshell --replicas=1
Local mTLS files missing	Deploy was interrupted before credentials were persisted	Extract from cluster secret openshell-client-tls (Step 6)
Container not found	Image not built	mise run docker:build:cluster (local) or re-deploy (remote)
Container exited, OOMKilled	Insufficient memory	Increase host memory or reduce workload
Container exited, non-zero exit	k3s crash, port conflict, privilege issue	Check openshell doctor logs for details
/readyz fails	k3s still starting or crashed	Wait longer or check container logs for k3s errors
OpenShell pods Pending	Insufficient CPU/memory for scheduling, or PVC not bound	openshell doctor exec -- kubectl describe pod -n openshell and openshell doctor exec -- kubectl get pvc -n openshell
OpenShell pods CrashLoopBackOff	Server application error	openshell doctor exec -- kubectl -n openshell logs statefulset/openshell
OpenShell pods ImagePullBackOff (push mode)	Images not imported or wrong containerd namespace	Check openshell doctor exec -- ctr -a /run/k3s/containerd/containerd.sock -n k8s.io images ls (Step 5)
OpenShell pods ImagePullBackOff (pull mode)	Registry auth or DNS issue	Check openshell doctor exec -- cat /etc/rancher/k3s/registries.yaml and DNS (Step 8)
Image import fails	Corrupt tar stream or containerd not ready

Full Diagnostic Dump

Run all diagnostics at once for a comprehensive report:

echo "=== Connectivity Check ==="
openshell status

echo "=== Container Logs (last 50 lines) ==="
openshell doctor logs --lines 50

echo "=== k3s Readiness ==="
openshell doctor exec -- kubectl get --raw='/readyz'

echo "=== Nodes ==="
openshell doctor exec -- kubectl get nodes -o wide

echo "=== Node Conditions ==="
openshell doctor exec -- kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{range .status.conditions[*]} {.type}={.status}{end}{"\n"}{end}'

echo "=== Disk Usage ==="
openshell doctor exec -- df -h /

echo "=== All Pods ==="
openshell doctor exec -- kubectl get pods -A -o wide

echo "=== Failing Pods ==="
openshell doctor exec -- kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded

echo "=== OpenShell StatefulSet ==="
openshell doctor exec -- kubectl -n openshell get statefulset/openshell -o wide

echo "=== OpenShell Service ==="
openshell doctor exec -- kubectl -n openshell get service/openshell

echo "=== TLS Secrets ==="
openshell doctor exec -- kubectl -n openshell get secret openshell-server-tls openshell-server-client-ca openshell-client-tls

echo "=== Recent Events ==="
openshell doctor exec -- kubectl get events -A --sort-by=.lastTimestamp | tail -n 50

echo "=== Helm Install OpenShell Logs ==="
openshell doctor  -- kubectl -n kube-system logs -l job-name=helm-install-openshell --=100

 
openshell doctor  --  /etc/rancher/k3s/registries.yaml

 
openshell doctor  --  -la /opt/openshell/bin/openshell-sandbox

 
openshell doctor  --  /etc/rancher/k3s/resolv.conf