pod-e2e

# pod-e2e — test a worktree against an isolated full-stack pod A worktree's full stack (backend API **and** frontend SPA) runs as **one process on one port** — exactly like a Docker container. The `kirocrew pod` CLI is the only interface you need: spin one up, get a `{base_url, token}` handle, test against it, tear it down and have the teardown VERIFIED. The live gateway is **never touched**. ## Quickstart — run the bundled e2e suite ```bash bash <app-skills-dir>/pod-e2e/scripts/pod-e2e.sh <worktree-name> --video ``` **Expected success output** — a `POD-E2E SUMMARY` ending like: ``` ✅ auth — GET /api/sessions → 200 with token, 403 without ✅ playwright — headless chromium loaded dashboard … result: 2 passed, 0 failed ARTIFACT_DIR=~/.kirocrew-pods/.e2e-artifacts/<worktree-name> ``` Exit code = number of failed phases (0 = all green) once phases run. Setup errors exit before any phase with their own codes (see the flag table below); a production-port SAFETY refusal exits 1. Then **look at the evidence**: `Read` the screenshots in that `ARTIFACT_DIR` (`fe-smoke.png`, plus any spec screenshots) to confirm the real UI rendered — not a 403/blank page. To smoke-test isolation after it finishes: ```bash kirocrew pod ls # should be empty (torn down) curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:5476/api/sessions # live plane still alive ``` ## The interface — pod tools if you are an agent, the CLI if you are a human **If you are an agent session, reach for the `pod_up` / `pod_down` / `pod_status` / `pod_ls` MCP tools before a shell.** A pod is a systemd `--user` unit, so every pod verb needs the systemd user bus. Whether your shell can reach that bus depends on the sandbox your session runs behind: under Kiro Crew's own sandbox it can (see `security.md`, "Scoped user-bus locator forward"), but behind an outer sandbox with its own user namespace it cannot, and `kirocrew pod up` then fails with `Permission denied`. `systemctl --user is-system-running` tells you which case you are in. The tools work either way -- the gateway holds the host bus and does the systemd part -- so they are the portable choice, and the only one on a host that denies you the bus. ``` pod_up {"worktree": "<wt>"} -> prose: port, base_url, token (+ttl) pod_status {"worktree": "<wt>"} -> status + port + health pod_ls {} -> every pod active on this host pod_down {"worktree": "<wt>"} -> stopped, HOME reclaimed ``` Provisioning is not one of them: a cold venv plus an SPA build is minutes of work, and a single blocking tool call for that dies to any timeout with no way to learn the outcome. An unbuilt worktree is refused with the CLI's own remedy in the message; run `kirocrew pod provision <wt>` or use the Dev Fleet page's Provision button, which streams its output. Everything after step 1 below -- curl against the handle, drive the SPA with Playwright, read the screenshots -- is unaffected: those are ordinary loopback HTTP, which a sandbox does not block. Only the systemd control path is walled off. The tools need the Dev Fleet app enabled; a disabled app refuses with `app_not_enabled`. The CLI is the same operations for a human at a terminal: ```bash # 1. bring the pod up, get a handle (JSON: base_url + token + port) kirocrew pod up <wt> --json # → {"name":"<wt>","status":"up","port":7958, # "base_url":"http://127.0.0.1:7958","token":"…","ttl":"2h"} # 2. test against the handle — full stack, ONE port: curl -s "$base_url/api/<anything>?token=$token" # backend API # open $base_url/?token=$token in Playwright # frontend SPA (same port) # 3. destroy it — deletes the HOME and verifies it is gone (nonzero if not), # live gateway untouched kirocrew pod down <wt> ``` Other verbs: `ls` (list running pods) · `status <wt>` · `token <wt>` · `url <wt>` · `logs <wt>` · `provision <wt>`. Run `kirocrew pod --help` for the full list. `token`, `url`, `logs` and `provision` have no tool of their own yet: `pod_up` already returns the token and url, and pod logs stay a human-facing read. **Isolation guarantees** (enforced by the pod runtime): own `KIROCREW_HOME`, own port, **no tunnel** (can't grab the real Slack identity), `--no-crons`, and cleanup on `pod down`. A pod can **never** collide with the live gateway and many can run at once. **Resource ceilings are Linux-only.** On Linux the unit sets cgroup `MemoryMax=4G` / `CPUQuota=200%`, which the kernel enforces. **On macOS there is no such ceiling and none is emitted** — macOS has no cgroups, and nothing it does offer bounds the total memory of a *process tree* or hard-caps CPU (`RLIMIT_AS` covers one process's address space, not resident memory, and the gateway spawns agent subprocesses that each get their own limit). So on a Mac a runaway pod can starve the machine; every other isolation property above still holds. **Teardown belongs to `pod down`, on both platforms.** It stops the service, waits for the process tree to drain, deletes the isolated HOME, and verifies the directory is gone — a HOME that survives is reported as a failure, never as zero residue. Nothing reclaims from a post-stop service hook: systemd would run one before the final kill of the pod's cgroup (racing the pod's own subprocesses) and on the stop half of a restart. So a pod that goes away WITHOUT a `down` (host crash, force-reboot, a raw `systemctl --user stop`) leaves its HOME behind on either OS; `pod ls` reports those (reclaim each with `pod down <name>`). This is also what makes a **seeded** pod home survive `systemctl --user restart` and `Restart=on-failure` instead of silently reverting to a blank instance. ## Bundled suite (quick path) ```bash bash <app-skills-dir>/pod-e2e/scripts/pod-e2e.sh <worktree-name> ``` Runs the bundled orchestrator end-to-end. Prints a `POD-E2E SUMMARY` ending in `ARTIFACT_DIR=<path>` and exits with the **number of failed phases** (0 = all green). Before any phase runs it exits with a setup code instead: | exit | meaning | |---|---| | `64` | bad arguments, bad worktree name, or unusable `--handle-json` | | `65` | no `kirocrew` CLI with `pod` on PATH, or the artifact dir escapes `.e2e-artifacts` | | `66` | the worktree checkout could not be resolved | | `67` | no usable CLI in the worktree venv (`kirocrew pod provision <wt> --venv-only`) | | `70` | the live-plane port could not be resolved (python3 missing) | | `1` | SAFETY refusal: the pod resolved to the production port (also a 1-failure run; read the summary) | Flags: | flag | effect | |---|---| | `--handle-json <path>` | run against a pod the CALLER started, from the handle in that file (see below) | | `--keep` / `--no-stop` | leave the pod running after tests (debug) | | `--api-only` | skip the Playwright phase (leaves a boot + auth check) | | `--fe-only` | accepted no-op — no test-suite phase exists to skip | | `--video` | record the session at 1080p → `.webm` + `.mp4` (finalization is time-capped) | | `--no-suppress-first-run` | let first-run modals appear (onboarding tests only) | ### Running the suite when you cannot reach the bus The script drives the pod with `kirocrew pod` verbs, and those need the systemd user bus, so on a host whose outer sandbox denies it the harness cannot start -- even though every phase after the first is ordinary loopback HTTP. `--handle-json` is that path: boot the pod with the `pod_up` tool, then build the JSON object shown below out of the values it reports and hand that file over. Do not paste the tool's own reply into the file: it is labelled prose, and `json.load` rejects it with exit 64. ```bash # pod_up {"worktree": "<wt>"} -> {"name": "<wt>", "base_url": ..., "token": ..., "port": ...} # pod_status {"worktree": "<wt>"} -> health # keep pod_up's name and write the fresh health code into that object, then: bash <app-skills-dir>/pod-e2e/scripts/pod-e2e.sh <wt> --handle-json "$H/handle.json" --video # ... and when you are done, pod_down {"worktree": "<wt>"} ``` The file is a JSON object you build: `name` is the `<wt>` you passed to `pod_up` (its reply names it but carries no `name` field), `base_url`, `token` and `port` are the values its reply prints, and `health` is the code `pod_status` reports: ```json {"name": "<wt>", "base_url": "http://127.0.0.1:7813", "token": "...", "port": 7813, "health": 200} ``` `name` must be a non-empty string exactly equal to the `<wt>` argument. The harness refuses a missing, non-string, empty, or different name before any phase runs; it never normalizes this field. Use exactly the worktree string you passed to `pod_up`. `base_url` must be an HTTP loopback address carrying an explicit port that is none of: the configured live-plane port (`KIROCREW_POD_LIVE_PORT`), its default `5476`, and the reserved `7777`. The default stays refused even when the variable names another port, because that variable is read from the harness's own environment: a deployment that sets it only in the gateway's unit would otherwise leave `5476` open here. Export it in the shell that runs the harness so the configured port is refused too. It and `token` may contain no whitespace or control characters. `token` must use the generated-token alphabet (`A-Z`, `a-z`, `0-9`, `-_.~+/=`), and `port` must be a canonical decimal integer from 1 through 65535 that matches the URL port. The harness validates these fields once, then runs from a canonical payload: `base_url` is rebuilt as `http://<loopback-host>:<port>` with no path, `port` is plain decimal, and the token is the validated value. No raw handle field reaches the production-port refusal, curl configuration, or Playwright. That is not shape pedantry: validating one representation and using another can point the harness at the live plane. The `pod_up` producer already emits the canonical URL, token, and port; add only the fresh `health` code from `pod_status`. In that mode the script calls no pod verb at all, and three things follow: - **The pod is yours, not the harness's.** It is never stopped on exit, whether the run passes or crashes. Call `pod_down` yourself. - **The health verdict is read, not measured.** The summary says `health — supplied by the caller`, and a stale handle means the run is judging a pod that has already died. Re-read it with `pod_status` immediately before the run, and pass what it reports rather than a remembered 200. - **The lifecycle ran on the gateway's build, not the worktree's.** The CLI path deliberately pins the worktree's own `kirocrew` so `pod up` / `pod down` exercise the branch under test. Nothing pins the gateway. So for a diff that changes pod lifecycle code itself, the CLI path is the one that tests it -- a green handle-mode run says the branch's *gateway* boots and renders, not that the branch's *pod code* works. - **`name` ties the handle to a worktree, not to a clone.** The check refuses a handle whose `name` is not the worktree you asked for, but a pod name is a worktree basename, and two clones on one host can each hold a worktree of that name. Run the harness from the clone whose pod you booted: with a handle from the other one, the specs and the artifact directory come from here while the requests go there, and the verdict is filed under the wrong checkout. Nothing in the handle, and nothing the pod serves, identifies a checkout today -- closing it needs `pod_up` to emit a machine-readable handle that carries one. A missing or malformed handle is refused before any phase runs, naming the field that is wrong. **It does not run the worktree's test suite.** There is no pytest phase, on purpose: the suite is ~62k tests that need no pod, CI runs it on the merge ref, and a full local fan-out on a shared box costs far more than the browser check this harness exists for. Run the tests your change actually touches yourself, in the worktree (see the `kirocrew-worktree-dev` skill), and let CI own the full suite. ## What each phase does 1. **up** — `kirocrew pod up <wt> --json`. If already active, reuses it (and won't stop it on exit). With `--handle-json` nothing is started at all: the handle IS this phase. Boots the worktree's own gateway with `--no-crons`, blank-seed DB, isolated HOME. If this phase fails with `gateway still starting after Ns` in `pod-up.log`, the gateway was alive but slower than the health-wait budget (default 90s): re-run with `KIROCREW_POD_HEALTH_SECS=<higher>` exported — the harness passes its environment through to the `pod up` it spawns. 2. **health** — polls `kirocrew pod status <wt> --json` until its `health` is 200/401/403 (≤60s). Deliberately not a bare `curl base_url/api/health`: a derived port is routinely held by another pod or by the live gateway, every gateway answers that path identically, so a 200 there proves only that *something* is listening. `pod status` reports the pod's OWN health and returns `-2` when the responder is provably somebody else's, which this phase reports as a port conflict naming `PORT=`. On timeout it dumps logs to `boot-fail.log` and aborts. 3. **auth** — proves auth: `/api/sessions` → 200 with token, 403 without. Token comes from `kirocrew pod up --json` output (no manual minting needed). 4. **Playwright** — `pod-playwright.py` (run with a Playwright venv + bundled chromium) loads `/?token=` headless, asserts the SPA rendered (screenshots `fe-smoke.png`), then exec's the optional `PLAYWRIGHT_SPEC` with a live authed `page` in scope. - If the Playwright interpreter is not executable, the FE phase **fails** — it does not skip. A run that captured zero screenshots must never report a green summary. `--api-only` is the one clean skip. - `--video` requires `ffmpeg` for mp4 transcoding; if absent, the `.webm` is kept but no `.mp4` is produced. - **Bounded, always.** The whole phase runs under `timeout` (`POD_E2E_PW_TIMEOUT`, default 600s) and each browser-teardown step under its own cap (`POD_E2E_TEARDOWN_TIMEOUT`, default 30s). Video finalization (`context.close()`) has been observed to block forever *after* a spec passed; on expiry the driver keeps every artifact, kills the browser tree, and exits, and the summary reports `playwright — TIMED OUT` as a distinct outcome. A recording that grew past 200MB is reported and left un-transcoded — for a short spec that size is itself a defect signal. The per-step cap uses `SIGALRM`, so on a platform without it the teardown degrades to unbounded and says so in the log (the harness is POSIX-only anyway); the phase-level `timeout` still applies. 5. **collect** — all logs + screenshots land in `~/.kirocrew-pods/.e2e-artifacts/<wt>/`. The Playwright driver appends its rows (`smoke`, `spec`, spec `record(...)` rows, `teardown`) to `verdict.jsonl` **as they are decided**; the up/health/auth verdicts appear only in the printed summary (and `playwright.log` is unbuffered), so a stalled or killed run still leaves a readable verdict. The file is truncated at the start of **every** run — including runs that skip the FE phase — so it can never show a previous run's rows. The rest of the artifact dir DOES persist across runs, so check timestamps before trusting an old screenshot. 6. **stop** — `kirocrew pod down <wt>`: stops the service, waits for its process tree to drain, deletes the isolated HOME, and verifies it is gone — a HOME that survives fails the command rather than being reported as zero residue. Skipped if `--keep` or if the pod was already up. ## The test manifest (`.pod-test.sh`) Optional per-worktree file declaring how THIS feature is tested. Searched at `<worktree>/.pod-test.sh` then `<worktree>/src/kiro_crew/.pod-test.sh`. The manifest is parsed **declaratively** (a `PLAYWRIGHT_SPEC=` line is extracted textually) — it is **never sourced or eval'd** on the host. ```sh # .pod-test.sh PLAYWRIGHT_SPEC=".pod-e2e/feature.spec.py" # frontend spec, relative to the manifest's dir ``` The resolved spec must live under `<worktree>/.pod-e2e/`: the driver refuses any other path ("outside allowed directories") and fails the run. A manifest under `src/kiro_crew/` therefore needs `PLAYWRIGHT_SPEC="../../.pod-e2e/<spec>.py"` or an absolute path. The spec runs only when the smoke phase passed. ### Trust model The **pod** isolates the gateway under test (own `KIROCREW_HOME`, own port, no tunnel, resource caps). The **harness** is not a sandbox: it runs the worktree's own gateway, and exec's that branch's `PLAYWRIGHT_SPEC`, as your user — exactly like building and running the checkout yourself. Only run pod-e2e against branches you would be willing to build and run locally. ### Playwright spec contract