helix-deploy

name

helix-deploy

description

Install, configure, upgrade and debug a Helix deployment — control plane via Docker Compose or the helix-controlplane Helm chart, and the sandbox node (Hydra container runner) that actually runs spec-task desktops and sandboxes. Covers install.sh flags, Hydra per-session Docker isolation and its host requirements, runtime configuration, LLM/GPU setup, health checks, logs, database access, and a symptom-to-cause troubleshooting table. Use when the user wants to install or self-host Helix, deploy to Kubernetes, set up a sandbox/Hydra runner, upgrade, or debug a control plane, sandbox or agent desktop that isn't working.

Installing and debugging Helix

A Helix deployment is two pieces:

1. Control plane — API, frontend, Postgres, vectorchord-kodit (RAG), searxng, chrome. Runs from Docker Compose or the helix-controlplane Helm chart.
2. Sandbox node — one privileged container per host running Hydra, the container runner that spawns every agent desktop and every user sandbox. Without it, spec tasks are created but never get a sandbox.

GPU runners are a third, optional piece — only needed if you want Helix to serve models itself instead of calling an OpenAI-compatible API.

Do you need a GPU? Only for streamed agent desktops. See GPU: what actually needs one before buying hardware.

Once it's up, drive it with helix-cli; prove it works with helix-e2e.

Install: control plane

curl -sL -O https://get.helixml.tech/install.sh
chmod +x install.sh
sudo ./install.sh --controlplane --api-host https://helix.example.com

Useful flags (./install.sh --help for the full list):

Flag	Effect
--controlplane	API + Postgres + friends in $INSTALL_DIR (default ~/helix) via Docker Compose
--sandbox	Sandbox/Hydra node — see below
--runner	GPU runner container
--cli	Just the helix binary
--code	Enable Helix Code (agent desktops, streaming). Loads the uhid kernel module. Documented as needing a GPU and --api-host — omit it if you only want headless sandboxes
--api-host <url>	Public URL. HTTPS on Ubuntu also installs and configures Caddy
--runner-token <tok>	Shared secret joining runners/sandboxes to the control plane
--privileged-docker	Hydra privileged mode — see the warning below
--openai-api-key / --anthropic-api-key / --together-api-key	Inference credentials
--vhost-tls-mode auto + --letsencrypt-email	Built-in TLS for project web services and sandbox previews
--helix-version <v> / --upgrade	Pin or bump the version
-y	Non-interactive

Manual Compose install:

git clone https://github.com/helixml/helix.git && cd helix
cp .env.example-prod .env
# edit: SERVER_URL, KEYCLOAK_FRONTEND_URL, POSTGRES_ADMIN_PASSWORD,
#       KEYCLOAK_ADMIN_PASSWORD, RUNNER_TOKEN, inference keys
docker compose up -d && docker compose ps

.env settings that matter most:

Variable	Purpose
SERVER_URL	Public URL — must match how users reach it, or OAuth and generated links break
KEYCLOAK_FRONTEND_URL	${SERVER_URL}/auth/
RUNNER_TOKEN	Joins runners and sandboxes. Never ship oh-hallo-insecure-token.
OPENAI_API_KEY / ANTHROPIC_API_KEY / INFERENCE_PROVIDER	Hosted inference
DYNAMIC_PROVIDERS	Bulk-register providers: groq:gsk_x:https://api.groq.com/openai/v1,…
HELIX_VERSION	Image tag; latest at https://get.helixml.tech/latest.txt
API_PORT	Host port for the API (default 8080)
HELIX_SANDBOX_RUNTIMES	Sandbox runtime catalogue — see Hydra below

The production docker-compose.yaml deliberately has no sandbox service. Sandbox nodes are run standalone by install.sh --sandbox. COMPOSE_PROFILES only selects a sandbox in the dev compose file, where the profiles are code-nvidia, code-amd-intel, code-software and code-macos — the code/code-amd names in the .env.example-prod comments are stale.

Upgrade:

sudo ./install.sh --controlplane --upgrade --helix-version 2.12.3
# or by hand: set HELIX_VERSION in .env, then
docker compose pull && docker compose up -d

docker compose restart does not pick up .env or image changes — use pull + up -d. Snapshot or pg_dump the database before a version bump.

Install: sandbox node (Hydra)

This is the piece people forget, and the reason spec tasks sit forever with sandbox_state: absent.

# same machine as the control plane, with streamed desktops (needs a GPU)
sudo ./install.sh --controlplane --sandbox --code \
  --api-host https://helix.example.com --runner-token "$RUNNER_TOKEN"

# no GPU on this host: drop --code and stay on headless runtimes
sudo ./install.sh --controlplane --sandbox \
  --api-host https://helix.example.com --runner-token "$RUNNER_TOKEN"

# dedicated GPU box joining an existing control plane
sudo ./install.sh --sandbox \
  --api-host https://helix.example.com --runner-token "$RUNNER_TOKEN"

The installer detects the GPU (NVIDIA / AMD / Intel / AWS Neuron / none), installs the NVIDIA container runtime if needed, and writes $INSTALL_DIR/sandbox.sh — a plain docker run of ghcr.io/helixml/helix-sandbox:<tag> that you re-run to restart or upgrade the node.

What that container gets, and why:

Setting	Why
--privileged	Docker-in-Docker. Non-negotiable.
--network helix_default	Reaches the API as http://api:8080 when co-located
HYDRA_ENABLED=true	Always on in production
HYDRA_PRIVILEGED_MODE_ENABLED	From --privileged-docker; off by default
RUNNER_TOKEN	Must match the control plane exactly
SANDBOX_INSTANCE_ID	$(hostname) — how the node identifies itself
MAX_SANDBOXES	Concurrent desktops on this host (installer writes 10)
-v sandbox-storage:/var/lib/docker	Nested dockerd image layers
-v hydra-storage:/hydra-data	Per-session dockerd data + shared BuildKit cache
-v sandbox-data:/data	Session workspaces
-v /var/run/sandbox:/var/run/sandbox	Socket the API shares with the node
--device /dev/uinput --device /dev/uhid + cgroup rules c 13:*, c 226:*	Virtual HID and DRI for the streamed desktop
-v /run/udev:/run/udev:rw	Device enumeration; the Helm chart mounts it too and the pod won't start without it
--restart=always	The node comes back with the host
SANDBOX_DATA_PATH, XDG_RUNTIME_DIR, HELIX_FRAME_EXPORT_PORT	Set by the generated script; rarely changed

MAX_SANDBOXES is what the node is told; the capacity the API reports back can differ (a node started with MAX_SANDBOXES=10 reported "max_sandboxes": 20). Read the reported value from helix api /sandboxes, not from your own env.

Host prerequisites the installer handles for you — replicate them if you deploy by hand:

• uhid kernel module loaded and persisted in /etc/modules-load.d/helix.conf (with --code). This is a host operation. Running it inside a container fails with FATAL: Module uhid not found in directory /lib/modules/<kernel> because there is no /lib/modules tree there — load it on the host, then pass the device in.
• inotify limits raised to fs.inotify.max_user_watches=1048576, fs.inotify.max_user_instances=1024 in /etc/sysctl.d/99-helix-inotify.conf. Zed burns thousands of watches per instance; the defaults run out after a couple of desktops. (The sandbox image's own init sets a lower 524288 — the installer's host-level value is the one that matters.)
• /hydra-data on a real filesystem, not an overlay. Docker's overlay2 cannot mount on top of overlay — but the failure is silent, not loud: the nested daemon logs failed to mount overlay: invalid argument, falls back to the vfs storage driver, and then reports Daemon has completed initialization. It starts. You get a node that looks healthy while burning disk at a multiple of normal and building at a fraction of normal speed. Liveness is the wrong check:

  docker exec helix-sandbox docker info | grep "Storage Driver"   # want overlay2, not vfs
  

How Hydra works

Hydra gives each agent session its own Docker daemon, so two tasks on the same host cannot see or reach each other's containers.

helix API ──RevDial──▶ hydra (in helix-sandbox)
                          │  docker exec / cp / inspect
                          ▼
              per-session dockerd  /var/run/hydra/{session}/docker.sock
                          │  own bridge hydra{N} → 10.200.N.0/24
                          │  own DNS server on the bridge gateway
                          ▼
              desktop container ◀─veth pair─▶ user's containers
              eth0 = streaming net          10.200.N.x
              eth1 = 10.200.N.254

Consequences worth knowing:

• An agent running docker compose up inside its desktop starts containers on its isolated bridge, and http://webapp:3000 resolves in that desktop's browser because Hydra runs a DNS server per bridge that queries that session's dockerd.
• Unknown names fall through to the sandbox's /etc/resolv.conf, so corporate DNS, private TLDs and VPN-reachable services keep working inside agent desktops.
• Session A has no route to session B's subnet. That's the isolation guarantee.

Privileged mode (HYDRA_PRIVILEGED_MODE_ENABLED=true, --privileged-docker) replaces all of that with the host's own Docker socket, bind-mounted at /var/run/host-docker.sock. Every session then shares one daemon and one network. It exists for Helix-in-Helix development — an agent that needs to run Helix itself, or k3s, or anything else wanting a real Docker. It removes tenant isolation: every user can see and control every other user's containers. Only enable it on single-tenant or development hosts.

GPU: what actually needs one

The GPU is for video encoding and rendering a streamed desktop, nothing else. Split the question by workload:

Workload	Needs a render node?
Control plane (API, Postgres, RAG, frontend)	No
Headless sandboxes (headless-ubuntu, node22, python313, custom images)	No — no compositor, no encoder, just a container
Spec tasks with --runtime headless-ubuntu	No
ubuntu-desktop runtime — streamed GNOME, screenshots, video	Yes
Serving models on Helix's own runners	Yes (that's a runner, not a sandbox)

So a CPU-only box is enough to run coding agents, provided you keep them headless. You lose the streamed desktop, spectask screenshot, spectask stream and the desktop MCP tools; the agent itself, its repo, its shell and the whole spec-task workflow are unaffected.

Check what a node reports, and what the deployment can therefore run:

helix api /sandboxes | jq -r '.[] | "\(.id) gpu=\(.gpu_vendor) render=\(.render_node) status=\(.status)"'
helix sandbox runtimes          # marks desktop runtimes unavailable when no host can stream

On a deployment with no render-capable host, helix sandbox runtimes says so, and asking for a desktop runtime — via helix sandbox create --runtime ubuntu-desktop or a spec task pinned with --runtime ubuntu-desktop — is rejected immediately rather than failing at placement.

Version note. Headless-without-a-GPU is helixml/helix#3035. On binaries before it, a CPU-only node reporting gpu_vendor: "none" / render_node: "SOFTWARE" is skipped by placement for headless work too, because one predicate gated both. The symptom is a node that shows status: online while nothing ever lands on it and every sandbox fails to place. On those versions a sandbox node needs a render node even for headless workloads.

install.sh --code is documented as requiring a GPU, and the sandbox installer prints Warning: No GPU detected. Sandbox may not work correctly. before setting GPU_VENDOR=none. For a GPU-less host use code-software in the dev stack (software rendering via x264enc) and expect low frame rates; the desktop path is not the reason to run Helix on such a host.

Sandbox runtimes

The catalogue of images the Sandboxes API can start is control-plane config, not code:

Env (on the API)	Default
HELIX_SANDBOX_RUNTIMES	headless-ubuntu=ubuntu:22.04|sleep infinity,node22=node:22-bookworm-slim|tail -f /dev/null,python313=python:3.13-slim|tail -f /dev/null
HELIX_SANDBOX_DEFAULT_RUNTIME	headless-ubuntu
HELIX_SANDBOX_ALLOW_CUSTOM_IMAGE	false — set true to let callers pass an arbitrary image

Format is name=image[|keep-alive-command], comma-separated. Adding Go or Rust runtimes is a config change, no rebuild. Spec-task desktop runtimes (ubuntu-desktop, headless-ubuntu) are separate and versioned by the node's heartbeat.

Hydra knobs

Env (on the sandbox node)	Purpose
HYDRA_ENABLED	Per-session dockerd isolation (default true)
HYDRA_PRIVILEGED_MODE_ENABLED	Share the host Docker socket instead (see warning)
MAX_SANDBOXES	Concurrent desktops on this host
HELIX_DISK_PRESSURE_PATHS	Filesystems watched for admission control (default /var/lib/docker,/hydra-data)
HELIX_DISK_PRESSURE_REFUSE_FREE_PCT	Refuse new sessions below this free % (default 2.0)
HELIX_DISK_PRESSURE_STOP_FREE_PCT	Stop running sessions below this free % (default 1.0)
HELIX_SANDBOX_APT_MIRROR	Internal APT mirror for air-gapped hosts
HELIX_EXPERIMENTAL_DESKTOPS	Space-separated extra desktops to pre-pull (sway, zorin, …)
GPU_VENDOR	nvidia | amd | intel | neuron | virtio | none

"Sandbox refused: /var/lib/docker at 1.3% free" is disk-pressure admission control doing its job, not a bug — free space or raise the threshold.

Verify Hydra end to end

# 1. Is a node registered, online and heartbeating?
helix api /sandboxes | jq -r '.[] | "\(.id) \(.status) gpu=\(.gpu_vendor) active=\(.active_sandboxes)/\(.max_sandboxes) privileged=\(.privileged_mode) seen=\(.last_seen)"'

# 2. Does the API know about runtimes?
helix sandbox runtimes

# 3. The real test — API → RevDial → hydra → nested dockerd → container
export HELIX_ORG=acme          # or pass --org to every command below
helix sandbox create --name hydra-smoke --runtime headless-ubuntu --ttl 300
helix sandbox exec sbx_01xxx -- bash -lc "uname -a && cat /etc/os-release | head -2"
helix sandbox delete sbx_01xxx

Keep the org consistent: --org is resolved per command, so creating in one org and exec'ing without --org gives a misleading 404 sandbox not found rather than an authz error.

Read the outcomes as three cases, not two:

• Step 1 empty (literal [], exit 0 — it exits 0 whether or not a node exists, so if helix api /sandboxes; then always passes; test the payload, e.g. [ "$(helix api /sandboxes | jq 'length')" -gt 0 ]): no node registered. Check RUNNER_TOKEN and the node's logs for RevDial control connection established. Also just wait: a fresh node pre-pulls the multi-GB desktop image before hydra starts, which took ~10 minutes in one measured run, and the pod reads READY 1/1 throughout because the probe only checks dockerd.
• Step 3 passes: Hydra is healthy end to end.
• Step 1 shows an online node but step 3 fails — the case people misdiagnose. The error reads no available sandbox host with the requested runtime, which sounds like "no node" and sends you back to the token you just verified. It usually means the node has not got the runtime's image: the catalogue in HELIX_SANDBOX_RUNTIMES is control-plane config, and what the node pulled is independent of it. A node that pre-pulled only desktop images has no ubuntu:22.04, node:22-bookworm-slim or python:3.13-slim. Check with:

  docker exec helix-sandbox docker images
  

helix spectask health checks the API, active agent sessions and the MCP endpoint. It does not check sandbox hosts — use helix api /sandboxes for that.

On 2.12.3 its API line is a false negative: it probes /api/v1/health, which 404s, so it prints ⚠️ Status: 404 against a perfectly healthy control plane — and exits 0 regardless. The paths that do answer are /health, /healthz and /api/health. Trust curl -s -o /dev/null -w '%{http_code}' $HELIX_URL/health over that line.

Kubernetes

Two published charts. Do not install from a git clone — Chart.yaml is generated at release time and a clone produces a sentinel version.

Prerequisites the section used to assume: helm and a cluster. kind stands one up in about 20 seconds and is enough for the whole control plane; scripts/kind_helm_install.sh in the helix repo automates the lot. jq is used by most one-liners here.

Resource requirements. A measured install of the full five-pod control plane on a cold host: ready 2m11s after helm install, then idling at ~1.8 GiB RAM, under 5% of 4 CPUs, ~5 GiB disk. It fits comfortably on a small VM — much smaller than the component list suggests.

helm repo add helix https://charts.helixml.tech
helm repo update

curl -o values.yaml \
  https://raw.githubusercontent.com/helixml/helix/main/charts/helix-controlplane/values-example.yaml
# set global.serverUrl, ingress, storageClass, postgresql
# AND replace controlplane.runnerToken — see the warning below

helm upgrade --install helix helix/helix-controlplane -f values.yaml \
  --set image.tag=$(curl -s https://get.helixml.tech/latest.txt)

kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=helix-controlplane --timeout=300s

Replace the runner token before you deploy. values-example.yaml ships controlplane.runnerToken: "your-secure-runner-token-here", and applying it verbatim gives you a cluster whose runner/sandbox join secret is a publicly known string. This is the Helm equivalent of oh-hallo-insecure-token, and it is easy to miss because the file's own comment reads like a placeholder you might get away with. Generate one: --set controlplane.runnerToken="$(openssl rand -hex 32)", or better, point controlplane.runnerTokenExistingSecret at a Kubernetes secret. Verify afterwards with kubectl exec deploy/<release>-helix-controlplane -- printenv RUNNER_TOKEN.

That label selector covers the whole stack, and 300s is enough in practice — a measured cold install had all five pods ready in 2m11s.

External Postgres:

helm upgrade --install helix helix/helix-controlplane -f values.yaml \
  --set postgresql.enabled=false \
  --set postgresql.external.existingSecret=my-pg-secret \
  --set postgresql.external.existingSecretHostKey=host \
  --set postgresql.external.existingSecretPortKey=port \
  --set postgresql.external.existingSecretUserKey=user \
  --set postgresql.external.existingSecretDatabaseKey=dbname \
  --set postgresql.external.existingSecretPasswordKey=password

Sandbox chart — this is the Hydra node on Kubernetes:

# the token secret's key must be literally "token"
kubectl create secret generic helix-runner-token --from-literal=token="$RUNNER_TOKEN"

helm upgrade --install helix-sandbox helix/helix-sandbox \
  --set sandbox.apiUrl=http://helix-helix-controlplane:80 \
  --set sandbox.runnerTokenExistingSecret=helix-runner-token \
  --set sandbox.maxSandboxes=10 \
  --set gpu.vendor=none \
  --set hydra.enabled=true

Three things in that command are easy to get wrong:

• gpu.vendor defaults to nvidia. Leaving it on a cluster without GPUs parks the pod in Pending forever with 0/1 nodes are available: 1 Insufficient nvidia.com/gpu. Set it to match the hardware; none is right for a CPU-only cluster.
• sandbox.apiUrl must name the real service. helm install helix helix/helix-controlplane creates helix-helix-controlplane, not helix-controlplane — the release name is prefixed. Check with kubectl get svc.
• runnerTokenExistingSecretKey defaults to token, so the secret must have a key of that name.

The node also needs /run/udev to exist on the host, or the pod never starts: MountVolume.SetUp failed for volume "udev": hostPath type check failed: /run/udev is not a directory.

Chart specifics that bite:

• securityContext.privileged: true is required (DinD). On OpenShift you also need the privileged SCC — see the openshift: block in values.yaml.
• Three PVCs are provisioned: dockerStorage → /var/lib/docker (50Gi), hydraData → /hydra-data (20Gi), workspaceData → /data (100Gi). hydraData must be a real volume; overlay-on-overlay breaks nested dockerd.
• Disk-pressure admission only watches the first two by default. To cover /data too:

  sandbox:
    extraEnv:
      - name: HELIX_DISK_PRESSURE_PATHS
        value: "/var/lib/docker,/hydra-data,/data"
  

• Default probes exec into the nested dockerd. On AKS with cgroup v2 they fail spuriously — disable them (--set probes.liveness.enabled=false) or override via a values file. Using --set to add an httpGet merges rather than replaces and yields an invalid spec.
• hydra.privilegedMode: true disables tenant isolation. Dev only.
• Per-cloud starting points ship as charts/helix-sandbox/values-{gke,eks,aks,bare-metal}.yaml.

scripts/kind_helm_install.sh stands the whole thing up on kind — the cheapest rehearsal. USE_LOCAL_HELM_CHART=1, USE_EXTERNAL_POSTGRES=1 and INSTALL_SANDBOX=1 toggle the variants. Read charts/helix-controlplane/UPGRADE.md before bumping a chart version.

Local development stack

cp .env.example-prod .env
./stack start                     # docker-compose.dev.yaml
docker compose -f docker-compose.dev.yaml ps

Ready means helix-api-1, helix-frontend-1 and helix-postgres-1 all Up and curl -s -o /dev/null -w '%{http_code}' http://localhost:8080 returning 200. Cold bring-up takes 5–10 minutes — connection-refused or Restarting early on means "still coming up". Poll; don't conclude failure.

The dev stack's sandbox is a compose profile, unlike production:

COMPOSE_PROFILES=code-nvidia ./stack start     # or code-amd-intel / code-software / code-macos

./stack stop                      # STOP_POSTGRES=1 / STOP_PGVECTOR=1 for data services too
./stack up <service>
./stack rebuild <service>
./stack lint
./stack test [./path/...]
./stack psql
./stack update_openapi
./stack build-sandbox             # Hydra + DinD image
./stack build-ubuntu              # desktop image (GNOME + Zed + streaming)
./stack build-zed release         # must be `release` on ARM
./stack help

Hot reload: the API rebuilds on save via air; the frontend is Vite HMR on 8081 behind 8080. Hydra does not hot reload — Air doesn't rebuild the binary that lives inside the sandbox container:

cd api && CGO_ENABLED=0 GOOS=linux go build -o /tmp/hydra-linux ./cmd/hydra
docker cp /tmp/hydra-linux helix-sandbox-nvidia-1:/usr/local/bin/hydra
docker compose -f docker-compose.dev.yaml exec -T sandbox-nvidia pkill -TERM hydra
docker logs helix-sandbox-nvidia-1 | grep "RevDial control connection established"

Desktop images and the settings-sync-daemon need ./stack build-ubuntu plus a new session.

Avoid ./stack start-tmux non-interactively, and never docker builder prune / docker system prune on a Helix dev machine — delete old image tags instead if disk is full.

Inference backend

A — external provider (no GPU). Set INFERENCE_PROVIDER, OPENAI_API_KEY, OPENAI_BASE_URL in .env (any OpenAI-compatible endpoint), restart the api service. Or register it live:

helix provider create -n my-vllm -u https://vllm.internal/v1 -f ./key.txt -m qwen3-32b

B — GPU runner. ./install.sh --runner --api-host <url> --runner-token <tok>, or run docker-compose.runner.yaml on the GPU box. No GPU locally? Run it remotely and reverse-tunnel: ssh -R 8080:localhost:8080 user@gpu-box. Confirm it appears under /dashboard.

Health checks

curl -s -o /dev/null -w '%{http_code}\n' $HELIX_URL      # 200 = up
helix version
helix organization list                                   # auth works end to end
helix api /sandboxes                                      # sandbox/Hydra nodes
helix sandbox runtimes
helix spectask health                                     # API + agent sessions + MCP
docker compose ps                                         # or: kubectl get pods
docker ps | grep helix-sandbox

Logs

docker compose logs --tail 100 api | grep -iE "error|failed|panic"
docker logs -f helix-sandbox                              # production sandbox node
docker logs helix-sandbox 2>&1 | grep -iE "revdial|hydra|dockerd"

# the per-session desktop containers live inside the sandbox's nested docker
docker exec helix-sandbox docker ps --format "{{.Names}}\t{{.Image}}"
docker exec helix-sandbox docker logs <container> 2>&1 | tail -100

# hydra itself, and the per-session dockerd sockets it hands out
docker exec helix-sandbox ps aux | grep -E "hydra|dockerd"
docker exec helix-sandbox ls -la /var/run/hydra/active/
docker logs helix-sandbox 2>&1 | grep "RevDial control connection established"

# kubernetes
kubectl logs -l app.kubernetes.io/name=helix-controlplane --tail=200
kubectl logs -l app.kubernetes.io/name=helix-sandbox --tail=200

In the dev stack the sandbox service is sandbox-nvidia (Linux GPU), sandbox (Linux no GPU) or sandbox-macos; in production it's a plain container named helix-sandbox.

Reading the output: hydra --socket /var/run/hydra/hydra.sock --socket-dir /var/run/hydra/active --data-dir /hydra-data is the runner itself. One dockerd on /var/run/docker.sock is the node's own nested daemon; additional [INNER-DOCKERD] processes are per-session daemons. In isolation mode each live session also has a socket under /var/run/hydra/active/ — in privileged mode that directory stays empty, because sessions share the host daemon instead. Desktop containers are named ubuntu-external-<session> or headless-external-<session>.

Database

docker exec helix-postgres-1 psql -U postgres -d postgres -c "SELECT count(*) FROM users;"
docker exec helix-postgres-1 psql -U postgres -d postgres -c \
  "SELECT id, name, status FROM spec_tasks ORDER BY created_at DESC LIMIT 10;"

Credentials differ by install method: Docker Compose uses database postgres, user postgres; the Helm chart uses database helix, user helix (psql -U postgres there fails with FATAL: role "postgres" does not exist). Git repositories live at /filestore/git-repositories/ inside the API container. Order spec tasks by created_at — there is no created column.

Treat direct SQL as read-only diagnostics; mutate through the API so orchestrator state stays consistent.

Troubleshooting

Symptom	Cause and fix
localhost:8080 refuses connections right after start	Still booting. Poll for several minutes.
API restarting in a loop	Postgres not ready or wrong credentials — docker compose logs api.
Login redirects somewhere broken	SERVER_URL / KEYCLOAK_FRONTEND_URL don't match how you reach it.
401 on every CLI call	Runner token instead of a user hl- key.
Spec task never gets a sandbox	No sandbox node. helix api /sandboxes — empty means none registered.
Node online but nothing ever places on it	Check gpu_vendor / render_node. Before #3035, none/SOFTWARE was excluded from placement even for headless work.
no sandbox host with a display/render node	Working as intended on a CPU-only fleet — use a headless runtime, or add a host with a GPU.
Node registered but status not online	Heartbeat stopped; check docker logs helix-sandbox.
Node never appears	RUNNER_TOKEN mismatch, or it can't reach HELIX_API_URL.
sandbox create hangs then fails	Nested dockerd not up. docker exec helix-sandbox docker info.
Node healthy but builds crawl and disk fills	Nested dockerd fell back to vfs because /hydra-data is on overlay. Check docker info | grep "Storage Driver" — it will be running, so liveness tells you nothing.
"refused: /var/lib/docker at N% free"	Disk-pressure admission control. Free space or raise the threshold.
Agent's docker compose up works but the browser can't reach the service	Desktop not bridged to the Hydra network — check hydra logs for veth/bridge errors.