coder-k3s

name	coder-k3s
description	Deploy, configure, manage, and troubleshoot Coder self-hosted dev environments on a k3s Kubernetes cluster. Use when: installing Coder on k3s, creating or updating workspace templates, debugging 502 Bad Gateway errors, fixing code-server connectivity, managing Coder CLI authentication, pushing Terraform workspace templates, configuring PostgreSQL backend, debugging workspace pod failures, setting up per-user namespaces, or integrating Coder workspaces with LiteLLM / GPU nodes.
argument-hint	install | template | workspace | troubleshoot | auth

Coder on k3s

End-to-end skill for running Coder (self-hosted dev environments) on a k3s cluster over Tailscale.

Cluster Context

• Coder server: mini1 (NodePort 31557), namespace coder
• Access URL: http://100.101.186.51:31557 (Tailscale direct) or SSH tunnel → http://localhost:8081
• SSH tunnel: ssh -L 8081:100.101.186.51:31557 bernard@mini1 -N
• PostgreSQL: coder-db pod in coder namespace on mini1
• Helm values: /home/bernard/coder/helm-values.yaml
• Templates dir: /home/bernard/coder/workspace-template/ (ai-workspace), /home/bernard/coder/workspace-template-minimal/ (minimal-ubuntu)

When to Use

• Installing or upgrading Coder via Helm
• Authenticating the coder CLI (token-based, non-interactive)
• Creating or pushing workspace templates
• Debugging workspace pod failures (502, connection refused, bad gateway)
• Configuring GPU vs CPU node scheduling in templates
• Setting up per-user namespace isolation
• Integrating workspaces with LiteLLM or other cluster services

---

Install & Upgrade

See ./references/install.md for full procedure.

Quick upgrade:

helm upgrade coder coder-v2/coder -n coder -f /home/bernard/coder/helm-values.yaml

---

CLI Authentication

Coder CLI requires a session token — browser auth doesn't work over SSH.

# 1. Open UI → Settings → Tokens → + New token
# 2. Run:
ssh bernard@mini1 "coder login http://100.101.186.51:31557 --token <token>"

Verify:

ssh bernard@mini1 "coder list"

---

Template Management

See ./references/templates.md for authoring guide.

Push an update

scp -r /home/bernard/coder/workspace-template bernard@mini1:/tmp/
ssh bernard@mini1 "coder templates push ai-workspace --directory /tmp/workspace-template/ --yes"

Update existing workspace to new template version

ssh bernard@mini1 "coder update <owner>/<workspace>"

List templates

ssh bernard@mini1 "coder templates list"

---

Workspace Storage — What Persists vs What Doesn't

Each workspace has a PVC mounted at /home/coder. Only this directory survives a workspace stop/start.

Location	Persists?	Notes
/home/coder/**	✅ Yes	PVC — survives pod deletion
~/.local, ~/.config, ~/.bashrc	✅ Yes	Inside /home/coder
System packages (apt install)	❌ No	Container filesystem — gone on stop
/usr/local/**, /opt/**	❌ No	Ephemeral container layer

Making installed tools persist

Option 1 — Install to home dir (immediate, no template change):

# Example: install a binary to ~/.local/bin (persists)
curl -fsSL https://example.com/install.sh | INSTALL_DIR=/home/coder/.local/bin bash
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc

Option 2 — Add to startup_script (reinstalls on every start, reproducible):

startup_script = <<-EOT
  # Pixi package manager — install once, cached on PVC
  if [ ! -f /home/coder/.pixi/bin/pixi ]; then
    curl -fsSL https://pixi.sh/install.sh | bash
  fi
  echo 'export PATH="/home/coder/.pixi/bin:$PATH"' >> /home/coder/.bashrc
  
  # System tools reinstalled on every start (fast with apt cache)
  sudo apt-get install -y -qq curl git vim python3-pip
EOT

Option 3 — Custom Docker image (fastest startup, most reproducible): Replace codercom/enterprise-base:ubuntu with a custom image that has your tools pre-baked.

---

Workspace Operations

# List all workspaces
ssh bernard@mini1 "coder list"

# Create workspace from template
ssh bernard@mini1 "coder create --template=ai-workspace --org=coder myworkspace"

# Delete workspace
ssh bernard@mini1 "coder delete <owner>/<workspace> --yes"

# Get workspace pod
kubectl get pods -A | grep coder- | grep -v '^coder '

# Exec into workspace pod
kubectl exec -n coder-<owner> workspace-<name> -- bash

restart vs update — important distinction

Command	Template version used	When to use
coder restart <ws> --yes	Same version as currently running	Bounce the pod (no template change)
coder update <ws>	Latest published template version	After pushing a template fix — does NOT accept --yes

After pushing a template fix with coder templates push, always use coder update (not coder restart) to pick up the change:

# Push fix
scp -r /home/bernard/coder/workspace-template-minimal bernard@mini1:/tmp/
ssh bernard@mini1 "coder templates push minimal-ubuntu --directory /tmp/workspace-template-minimal/ --yes"

# Apply to a running workspace (no --yes flag available)
ssh bernard@mini1 "coder update <owner>/<workspace>"

Monitor until healthy:

ssh bernard@mini1 "coder logs <owner>/<workspace> --follow"
# Confirm: HEALTHY: true
ssh bernard@mini1 "coder list"

---

Troubleshooting

See ./references/troubleshoot.md for all known issues.

Quick checks:

# Check startup script log
kubectl exec -n coder-<owner> workspace-<name> -- tail -50 /tmp/coder-startup-script.log

# Check if code-server is listening
kubectl exec -n coder-<owner> workspace-<name> -- ss -tlnp | grep 13337

# Check coder server logs
kubectl logs -n coder -l app.kubernetes.io/name=coder --tail=50