setting-up-devbox

name

setting-up-devbox

description

Guide a PostHog engineer through spinning up, connecting to, running commands on, and mirroring local code to a remote devbox (a Coder workspace running the full PostHog stack). Use when asked to set up a devbox, start or connect to a devbox, configure remote dev, get gh CLI / Claude Code authed on a devbox, run a command on a devbox, sync a local checkout so you can edit locally while the stack runs remotely (devbox:sync), or diagnose why a devbox command fails. Covers the tailnet prerequisite, hogli devbox commands, Coder user secrets for auth, one-way local->remote sync via mutagen, and verifying with devbox:exec. How each dev personalizes their box is left to them.

Setting up a PostHog devbox

A devbox is a Coder workspace running the full PostHog stack on an EC2 instance, managed through hogli devbox:* (the only supported interface — drive those commands, don't reimplement them). It ships ready to use: the repo cloned at ~/posthog, the stack pre-warmed, and Claude Code installed. This skill gets a dev connected and working; how they personalize beyond that is their choice, not something to push.

Prerequisite: tailnet access (the thing people miss)

The devbox control plane lives inside a private VPC reachable only over Tailscale. Two things have to be true, and they fail in ways that look identical at the reachability check — every hogli devbox:* command dies there, and it is neither an auth nor an install problem, so no amount of re-running devbox:setup fixes it:

1. You are signed into the posthog.com tailnet. PostHog has several tailnets — dev, prod-us, prod-eu and internal exist for CI runners and subnet routers, and none of them route you to devboxes. Humans want posthog.com and nothing else.
2. Your email is in group:engineering in tailnet-policy.hujson (posthog-cloud-infra), the ACL that grants the route to the Coder control plane (10.70.0.1:443).

Check the tailnet first — it's cheap, and it's the more common of the two:

tailscale switch --list                 # the active tailnet is marked; you want posthog.com
tailscale switch posthog.com            # if you've signed into it before
tailscale logout && tailscale login     # otherwise — pick posthog.com at the tailnet picker
# on macOS when `tailscale` isn't on PATH:
/Applications/Tailscale.app/Contents/MacOS/Tailscale switch --list

Being on the wrong tailnet is sticky and invisible: the picker only appears at first sign-in, so someone who clicked past it a year ago has been on dev ever since and was never asked again. In the GUI it's Add account → sign in as usual → select posthog.com. Suspect this whenever a dev says "it used to work on my other laptop" or is on a new machine.

If the tailnet is right and hogli devbox:doctor still reports the control plane unreachable, the fix is a PR adding the user to group:engineering in tailnet-policy.hujson (then ask Team DevEx if still blocked). Diagnose both of these before touching anything else.

Control plane unreachable with a DNS cause?

When doctor shows [ok] Tailscale connected but fails reachability with a DNS cause (DNS lookup for coder.dev.posthog.dev failed), the name is never reaching the internal resolver. In order:

1. Wrong tailnet (above). The other tailnets have no route to dev.posthog.dev, and this is what it looks like. Rule it out first.
2. MagicDNS off. "Use Tailscale DNS" in the client's DNS settings is what points the machine at the internal resolver; without it nothing internal resolves.
3. Stale resolver upstream. MagicDNS on and names still failing, but tailscale ping <internal-ip> pongs and the service loads over its raw IP? That's the machine's router/ISP resolver. Adding 8.8.8.8 or 1.1.1.1 to the host's DNS settings has fixed this for several people.

To confirm it's resolution rather than the grant: dig coder.dev.posthog.dev @10.90.0.2 answering while the system resolver fails proves the name exists and only the resolution path is missing.

Do not reach for an exit node. Devboxes are reached as tailnet peers, so an exit node is not required for devbox access — it just routes all of the dev's traffic through infra, which is noticeably slower, and it masks whichever of the three causes above is the real one. Don't suggest /etc/hosts or /etc/resolver workarounds either: they hardcode internal ELB IPs that rotate.

Full write-up, kept current by Team Cloud Foundations: wiki.posthog.com/access/vpn.

Workflow

1. Check state — hogli devbox:doctor

hogli devbox:doctor          # read-only: tailnet access, reachability, auth, ssh config, saved setup

A safe probe — it never prompts or mutates host config (unlike devbox:setup). It names the active tailnet on its own line (Tailnet: … (need posthog.com)) and, when that's the problem, says so outright (Cause: Signed into the 'dev' tailnet, not 'posthog.com'.) — trust that over any other symptom. If it flags the control plane unreachable, resolve the tailnet (and then the ACL grant) before anything else. For more detail: hogli devbox:list (your boxes), hogli devbox:status (state, template freshness), hogli devbox:secret:list (secret names only).

2. One-time local setup — hogli devbox:setup

Interactive: checks Tailscale + Coder reachability, installs and authenticates the coder CLI (plus the pinned mutagen binary that backs devbox:sync), and writes the SSH host entries that devbox:ssh/devbox:exec rely on. It then offers git identity, git signing, a dotfiles repo, and your Claude token — all optional; --skip-* anything you don't want. Re-run one step with its flag, e.g. hogli devbox:setup --configure-git-signing.

3. Start and connect — hogli devbox:start

hogli devbox:start           # create or resume your box
hogli devbox:ssh             # shell in
hogli devbox:open --vscode   # or --cursor / --web
hogli devbox:stop            # when done — preserves disk, stops billing

4. Fast QA and agent resume

Before recreating sync, restarting PostHog, or making a new devbox, check whether the existing box is already usable:

hogli devbox:status
hogli devbox:exec -- bash -lc 'cd ~/posthog && git status --short --branch && git rev-parse --short HEAD'
hogli devbox:sync --status
hogli devbox:exec -- bash -lc "curl -sf -o /dev/null -w '%{http_code}' http://127.0.0.1:8010/"

Use -n <name> on each command for a labeled box. If the forwarded app serves the intended branch/SHA, the target route loads, and route-critical APIs work, keep going and note unrelated degraded units rather than chasing perfect all-process health. When startup time matters, record rough timings for devbox start/resume, sync readiness, first route response, and first target-route load.

For agent-managed startup, hogli devbox:start --start-app is the supported path for new/stopped boxes. The flag is sticky and starts the regular PostHog stack in the background. If the box is already running without the app, start it inside the box:

hogli devbox:exec -- bash -lc 'cd ~/posthog && ./bin/hogli up -d -y'

Use the target route, route-critical APIs, and process-specific phrocs checks as the readiness gate.

5. Auth, if you want it (optional)

To have gh or Claude Code authenticated on the box, store the token once as a Coder user secret. It's injected as an env var into every box you start, so you set it once rather than per box:

hogli devbox:secret:set GH_TOKEN --env GH_TOKEN
hogli devbox:secret:set CLAUDE_CODE_OAUTH_TOKEN --env CLAUDE_CODE_OAUTH_TOKEN
# also supported: ANTHROPIC_API_KEY, OPENAI_API_KEY, OP_SERVICE_ACCOUNT_TOKEN, AWS_CREDENTIALS (--file)

Authing gh / Claude on a devbox is fine — that's what these are for. Set the value from --file or the hidden prompt; never paste a token into a command line or into this conversation. Restart a running box to pick up a newly set secret.

6. Make it yours — your call

The box is usable as shipped; personalize it however suits you, or not at all. Two supported paths, neither required, don't push one over the other:

• Tweak the box directly — devbox:ssh in and install tools, add aliases, clone repos. Changes under /home survive stop/start and template updates, but a devbox:destroy (or a brand-new box) starts fresh.
• A dotfiles repo — if you'd rather keep portable, version-controlled config that re-applies to every box: hogli devbox:setup --configure-dotfiles points the box at your dotfiles_uri, and Coder clones it (running an executable ~/dotfiles/install.sh if present) on each start.

7. Run commands on the box — hogli devbox:exec

devbox:exec runs one command over SSH and propagates its exit code — handy for scripts, agents, and quick checks without opening a shell:

hogli devbox:exec -- bash -lc 'gh auth status'
hogli devbox:exec -- bash -lc 'cd ~/posthog && git status'
hogli devbox:exec -n api -- bash -lc 'uname -a'    # -n targets a labeled box

Wrap commands in bash -lc '...': a non-login shell doesn't reliably source ~/.bashrc/~/.zshrc, so a bare gh auth status can report "command not found" for anything on a login-shell PATH (e.g. ~/.local/bin) — a false negative. The login shell also keeps the exit code trustworthy, so && chaining and if checks work. Use -- to separate hogli's flags from the command's own.

devbox:exec is not side-effect-free: like every devbox:* command it runs the reachability check first, which on Linux may sudo tailscale set --accept-routes and prompt for a password. Run hogli devbox:setup once interactively so routes and SSH config are in place before an agent drives devbox:exec unattended.

Editing locally, running on the box — hogli devbox:sync

When you want your fast local checkout to stay the place you edit but the heavy stack (hogli up) to run on the box, hogli devbox:sync mirrors your repo onto the box over mutagen, one-way: local is the source of truth, nothing comes back. Reach for this in an agentic loop — edit with your normal local tools, let the mirror carry each change, and drive the remote stack with devbox:exec — instead of committing and pushing every iteration or editing over Remote-SSH.

hogli devbox:start                                   # the box must be running first
hogli devbox:sync                                    # create the mirror (idempotent: re-run just reports status)
# edit files locally — changes propagate within seconds
hogli devbox:exec -- bash -lc 'cd ~/posthog && pnpm --filter=@posthog/frontend typescript:check'
hogli devbox:sync --status                           # watching / paused / conflicts
hogli devbox:sync --terminate                        # tear the mirror down when done

Before relying on sync, verify the remote branch/SHA and devbox:sync --status. If the box already matches and sync is watching without source-file conflicts, do not recreate it for cleanliness. Source-file conflicts block reliable QA until resolved; box-local config conflicts such as .env can be acceptable when the tracked source is clean.

The non-obvious parts:

• It runs on your machine and pushes to the box — not the reverse. Don't invoke it through devbox:exec. It mirrors whichever checkout you run it from (it walks up from the cwd for hogli.yaml + .git), so run it from the repo root you are editing — including a /wt worktree.
• one-way-safe preserves remote-only files. The AMI's prewarmed node_modules, venv, and target/ are never deleted — they aren't in your local checkout and the mode leaves remote-only content alone. Lockfiles do sync, so the box reconciles deps on its next start.
• The first sync of a feature branch conflicts per diverged file. The AMI is always on master; every file your branch changed relative to the box's master surfaces as a conflict in --status. That is expected one-way-safe behavior, and it is per-path — non-conflicting files (including brand-new ones) still sync. Resolve a path, or check the matching branch out on the box, only if you specifically need that file mirrored.
• Don't also edit those files on the box. Editing over Remote-SSH while the mirror is live fights the local source of truth; devbox:open --vscode|--cursor warns when a sync is active for exactly this reason.

The packaged ignore defaults are seeded once to ~/.hogli/mutagen.yml and never overwritten — it is yours to tweak. If a newer hogli ships updated ignore defaults, rm ~/.hogli/mutagen.yml and re-run devbox:setup to pick them up.

Persistence & multiple boxes

• devbox:stop → devbox:start and template/AMI updates preserve /home (the instance is stopped, not terminated). A devbox:destroy wipes it — intentional, so don't keep anything irreplaceable only inside a box.
• You can run more than one box. Box-local changes don't carry between them; user secrets do (user-scoped), and a dotfiles repo does if you use one. That's the practical reason to reach for those if you find yourself re-doing setup — but it's a choice, not a requirement.

Gotchas

• Never echo secret values into the transcript, logs, a PR, or a command line. devbox:secret:set reads from a hidden prompt or --file; secret:list shows names only. Keep it that way.
• Secrets need a restart. A new or changed secret only reaches boxes started afterward — hogli devbox:restart to pick it up on a running box.
• devbox:exec/devbox:ssh need devbox:setup to have run (it writes the coder.* SSH host config). Without it they fail at connection; devbox:doctor shows whether SSH access is configured.
• code-server (browser IDE) has no SSH agent forwarding, so commit signing via a forwarded key won't work there — use VS Code Desktop / Cursor / JetBrains (SSH-based) when you need to sign.