fray-artifact-e2e

# fray-artifact-e2e — verify on the promoted artifact, not the dev stack Fray serves a PROMOTED ARTIFACT from `~/.fray/builds/<digest>/runtime/`, never the source checkout (see the `fray-servers-run-promoted-artifacts` memory). The `adhoc-cdp` isolated stack runs `startServer({dev:true})` — SOURCE. So a green `adhoc-cdp` proves the dev path, NOT the shipped build: bundling, detached-daemon emission (`build-runtime.mjs` writes each `DETACHED_DAEMON_ENTRIES` file beside `index.js`), and executable/PATH resolution all differ. A real regression class (e.g. the Claude broker's daemon needing an ABSOLUTE claude path — a bare `"claude"` crashes it before it publishes its record, so every dispatch times out "did not become ready") is invisible to dev-source gates and to static checks (`node loads the file`, `the daemon publishes standalone`). Only launching the artifact + dispatching catches it. ## The recipe ### 1. Launch a real promoted-artifact instance, isolated `fray-dev` (= `nub … src/index.ts`) builds/promotes an artifact from CURRENT SOURCE and runs it. Two gotchas: - **You are probably running inside a fray worker**, so your env has `FRAY_DEV_CHILD=1` and `FRAY_LAUNCH_*`. Left set, `fray-dev` takes the CHILD path and dies `"Fray control-plane child has no live matching project launch owner"`. Strip them. - Use a **throwaway git repo** as the project (isolated `~/.fray/projects/<id>`), **real HOME** (so Keychain claude auth works — a sandbox HOME fails `AUTH_REQUIRED:claude`), and force the feature on. ```bash REPO=$(mktemp -d /tmp/fae-XXXX); (cd "$REPO" && git init -q . && git commit -q --allow-empty -m init) cd "$REPO" && exec env \ -u FRAY_DEV_CHILD -u FRAY_DEV_PORT -u FRAY_DIRECT_SUPERVISOR -u FRAY_LAUNCH_OWNER_TOKEN \ -u FRAY_LAUNCH_PROJECT_DIR -u FRAY_LAUNCH_PROJECT_ID -u FRAY_LAUNCH_STATE_DIR -u FRAY_LAUNCH_IDENTITY_SCOPE \ -u FRAY_LAUNCH_TMUX_SOCKET -u FRAY_LAUNCH_TMUX_SOCKET_MANAGED -u FRAY_PERM_DIR -u FRAY_SCRIPTS_DIR \ -u FRAY_SOURCE_COMMAND -u FRAY_STABLE_ARTIFACT -u FRAY_STABLE_WEB_DIST -u FRAY_UI_THREAD -u FRAY_WORKER_PLUGIN_DIR \ FRAY_CLAUDE_BROKER_BRIDGE=1 fray-dev ``` Run it with Bash `run_in_background: true`. First launch BUILDS the artifact (esbuild runtime + web build, ~30-90s). Wait for `requested Fray in your default browser — http://127.0.0.1:<PROXY>` (the proxy port; the child API logs `server on http://127.0.0.1:<CHILD> (prod)` — `(prod)` confirms it's the artifact, not dev). Drive the PROXY url. ### 2. Dispatch through the real RPC surface Use `scripts/lib/rpc-client.mjs` (never hand-rolled fetch — queries are `GET /rpc/<name>?input=`, mutations `POST`, every response is `{result}`/`{error}`): ```js import { createRpcClient } from ".../scripts/lib/rpc-client.mjs" const api = createRpcClient(`http://127.0.0.1:${PROXY}/`); await api.waitForHealth() const { slug, sessionId } = await api.mutate("dispatch", { prompt: "Reply with exactly HELLO-OK then stop." }) // poll for the agent's reply — this is the real end-to-end signal, not a status field for (let i=0;i<40;i++){ await sleep(3000) const tr = await api.query("threadTranscript",{slug}); if (JSON.stringify(tr).includes("HELLO-OK")) break } ``` For a worker-environment change, dispatch a prompt that uses the real surface — a `fray:haiku` sub-agent, the fray MCP (`mcp__fray__spawn_thread`), chrome-devtools — and assert the sub-agent's marker token routes back. A plain "write a file" proves almost nothing. ### 2b. A HAPPY-PATH PROMPT IS NOT A TEST — drive the ugly shapes **This is the step that has actually failed.** 2026-07-26 a spike shipped with a promoted-artifact e2e, a real-Chrome screenshot, a sub-agent round-trip, a kill-the-server survival test and 2150 green unit tests. Every dispatched prompt was a toy: *"reply with exactly HELLO-OK"*, *"write a marker file"*, *"count to 5"*. Within a day a real orchestrator thread and every one of its sub-agents were destroyed by a real agent running `printf '\033[31m…'` — an ANSI escape hit a strict protocol validator, the validator threw, and the daemon's pump read that as the session ending. A toy prompt cannot find that, and no amount of MORE toy prompts would have. The risk in a provider integration is not in the control flow you wrote; it is in the **values the provider sends back**, and a clean prompt only ever produces clean values. So the workload has to be hostile ON PURPOSE: - **control bytes and ANSI escapes** in a tool argument (`printf '\033[31mX\033[0m'`) — the shape that actually killed a thread; - **output far past the size bounds** in `claude-agent-sdk-protocol.ts` (128 KB event text, 64 KB JSON) — `head -c 400000 /dev/zero | tr '\0' 'x'`; - **non-UTF-8 / binary** bytes (`head -c 2048 /dev/urandom | base64`); - **unicode the validators single out** — bidi overrides, zero-width joiners, astral plane, combining marks; - **a long tool chain and a live sub-agent**, so turn bracketing and liveness are exercised together. `packages/server/src/backend/_live_broker_hostile.mts` is exactly this workload against a real session; run it (or extend it) rather than re-deriving the list. When you add a bound to the protocol, add a case. The assertion that matters is not "the reply was correct" — it is **"the session is still alive and still answering afterwards"**, checked against the daemon's own diagnostics log for a `lifecycle:crashed` record. A thread that dies quietly is the failure mode; a wrong answer is a much smaller problem. ### 3. Screenshot in a real browser tab The `chrome-devtools` MCP is the reliable path here (`scripts/shot.mjs`'s puppeteer Chrome tends to hang in a worker env). `new_page` the proxy url → `wait_for` the thread text → `take_screenshot`. **Write the shot under a chrome workspace root** (e.g. the fray repo dir), NOT `/tmp` (access denied); then move it to a trusted embed root (`~/Screenshots`, the project dir, or `os.tmpdir()`) for the handoff and clean the repo. The board only renders threads when the project has a `.fray/` dir — `mkdir -p $REPO/.fray/threads` if it's otherwise empty. ### 4. Clean up Kill the instance by its EXACT proxy+child PIDs (`lsof -nP -iTCP:<port> -sTCP:LISTEN -t`, never broad `pkill`), and `rm -rf` the throwaway repo + its `~/.fray/projects/<id>`. A fray server retains its artifact snapshot for its process lifetime, so the *in-app restart button* does NOT pick up a new build — only a full process kill + relaunch re-promotes from current source.