archie-e2e

name

archie-e2e

description

Boot a live Archie instance from the current branch, drive E2E scenarios headlessly through the archie-debug MCP (nonce → create_task → wait_for_task → approve), capture per-scenario evidence files, and tear down cleanly. Use for Forge Stage 4 QA, verifying a branch against a live instance, running the basic or edit-mode E2E scenario, or when asked to "run the e2e harness", "boot archie for testing", or "verify this AC live".

archie-e2e — boot-from-branch E2E harness

Verify acceptance criteria against a live Archie instance booted from the branch under test. The lifecycle is four steps, each with one documented invocation:

1. Boot — npx tsx tools/e2e/boot.ts builds and starts the instance detached, waits for /health.
2. Drive — execute scenario recipes through the archie-debug MCP (no scripts; you make the MCP calls).
3. Evidence — pipe a per-scenario payload to npx tsx tools/e2e/evidence.ts, which validates and writes the JSON + markdown pair.
4. Teardown — npx tsx tools/e2e/teardown.ts stops everything and verifies no project containers remain.

Prerequisites

Running in a cloud sandbox / CI behind a TLS-intercepting proxy (e.g. Claude Code on the web)? See docs/guides/e2e-in-cloud-sandbox.md for the cold-start setup (CA trust, key mapping, bind-mount ownership, setsid for long-running processes). The steps below assume direct internet egress.

• Docker (Desktop or daemon) running locally; docker compose version works.
• .env at the repo root with a non-empty ANTHROPIC_API_KEY. No Slack tokens are needed — scenarios enter through the CLI/API channel.
• The archie-debug MCP available (registered in .mcp.json as npx tsx tools/debug-mcp/server.ts). The harness consumes it as-is and never modifies tools/debug-mcp/.
• For the edit-mode and merge-approval scenarios: at least one configured engineering repo in the workdir (see the recipes' prerequisite notes).
• macOS Docker Desktop caveat: a wedged docker-credential-desktop helper can stall docker compose up --build during registry auth — upstream of the harness's bounded wait, so boot appears to hang before any diagnostics. Verify with docker-credential-desktop list </dev/null; if it hangs, point DOCKER_CONFIG at a scratch dir without credsStore for the run (leave your real ~/.docker/config.json untouched). The scratch config also drops the desktop-linux context, so additionally set DOCKER_HOST=unix://$HOME/.docker/run/docker.sock or every docker command will fail to reach the daemon (observed in the 2026-07-05 QA run).

Rollback: delete .claude/skills/archie-e2e/ and tools/e2e/ — the harness has no side effects and nothing else references them (plus one e2e-evidence/ line in .gitignore).

1. Boot

npx tsx tools/e2e/boot.ts [--timeout-seconds N]

The boot runs a strict ordered sequence; each step gates the next, so no failure ever falls through into the poll loop:

1. Preflight (mandatory, first): .env exists at the repo root and ANTHROPIC_API_KEY is non-empty. Failure exits non-zero naming the missing item, with no compose invocation at all.
2. Port preflight: the target port is probed before any compose invocation (skipped when ARCHIE_URL is set explicitly — that target is the operator's to manage). Free → proceed. Published by this project's own archie (checked against the container's published ports, not mere existence) → docker compose down first and boot fresh on the same port (the harness never adopts a running instance — it may be stale code; every boot builds from the current checkout). Held by anything else (a foreign archie from another worktree, an unrelated service) → a free port is auto-picked and the boot relocates there, leaving the squatter untouched.
3. Compose up, exit code trapped: docker compose up --build -d (detached — npm run docker:dev is foreground, don't use it here), with PORT and GIT_SHA passed in the compose environment. A non-zero exit prints the diagnostics block and exits non-zero before a single /health poll — a failed build never spends any of the wait cap.
4. Health poll: GET {baseUrl}/health every 5s until 200. Bounded by a default cap of 600s, overridable via --timeout-seconds or E2E_BOOT_TIMEOUT_SECONDS (flag beats env beats default). Each tick also checks docker compose ps --format json: an archie container that is exited, dead, restarting (crash loop), or missing fails immediately instead of waiting out the cap.
5. Checkout attestation: /health must report the git_sha the boot passed (git rev-parse HEAD, -dirty suffix on an unclean tree). A healthy instance reporting anything else fails the boot — positive proof the instance was composed from the checkout under test, not a stale build or another checkout. Quote the Attested: instance composed from <sha> line in evidence.

On success it prints the resolved base URL, the /health body, and a final ARCHIE_URL=http://localhost:<port> line. When the port was auto-picked, the driver MUST target that URL — and note the MCP constraint: the archie-debug MCP resolves its URL once at spawn (ARCHIE_URL → PORT env → PORT from .env → http://localhost:3000), so an MCP already attached to your session (spawned before boot) CANNOT be retargeted by exporting a variable afterwards. Either drive via a fresh stdio client you spawn yourself with ARCHIE_URL in its environment (ARCHIE_URL=... npx tsx tools/debug-mcp/server.ts, as the QA runner does), or restart the session/MCP so it picks the value up. On the default port none of this matters — the resolutions already agree.

Failure diagnostics (printed on any compose-up or poll-phase failure, always followed by a non-zero exit): the docker compose ps table plus the last 100 lines of the archie container's logs (docker compose logs --no-color --tail=100 archie). Read the log tail first — missing env keys, plugin clone failures, and crash loops all surface there.

Boot time observations: cold --build boot: ~45s build + ~2min to healthy (M-series Mac, 2026-07-05 QA run of forge run pr-167-mcp-file-bridge); warm boot (image cached): not yet measured. Tuning guidance: set --timeout-seconds to roughly 2× the observed cold time (~330s) for CI-like runs, and near the warm time for tight iteration once measured. The generous 600s default remains a safe ceiling.

2. Drive scenarios (archie-debug MCP recipes)

Scenarios are agent-driven: you call the MCP tools directly, following these recipes. The recipe names below are canonical and must be used verbatim as the evidence files' scenario field.

Constraints (all recipes):

• Serial only — run one scenario at a time; parallel runs are unvalidated.
• Nonce window — wait_for_task's nonce scan covers only the 25 most-recent tasks, so on a long-lived instance old nonces fall out of the window; remedy: fresh boot before a scenario session.
• Mint each nonce fresh: E2E-$(openssl rand -hex 4).

Recipe: basic-nonce

A read-only question reaches completed with an observable PM reply. Verifies the create → correlate → wait → assert loop (brief AC2).

1. Mint NONCE=E2E-$(openssl rand -hex 4).
2. create_task with the nonce embedded in the message, e.g. "[${NONCE}] What agents are configured in this instance? Reply with a short list and do not modify anything."
3. wait_for_task(nonce: NONCE) — first call correlates the nonce to a task id and returns TASK=<id>.
4. While STATE=pending: call wait_for_task(task_id: <id>, cursor: <CURSOR from the previous call>) again — each call waits up to ~45s server-side and returns a resumable CURSOR.
5. Terminal: assert STATE=completed, and a PM_REPLY line was observed (or fetch get_events(task_id) and assert a message event with data.from === 'pm-agent').
6. Excerpts for evidence: get_log(task_id, tail: 40) for the knowledge log (must contain the nonce), get_events(task_id) for task:created … task:completed.

Recipe: edit-mode-approval

A change request against a configured repo trips the edit-mode gate, is approved via the API path, and proceeds to completion (brief AC3).

Prerequisite: the workdir has at least one configured engineering repo (the PM only requests edit mode when a code change targets a repo the plugin config declares). If the prerequisite is unmet, do NOT fake a pass — report the scenario as BLOCKED, stating: what is missing (no configured engineering repo in the workdir), what was attempted, and the remedy (configure a repo via the plugins setup, reboot, re-run). A BLOCKED scenario produces no evidence file; it appears in the run report only.

1. Mint a fresh nonce as above.
2. create_task with a small, real change request against a configured repo, e.g. "[${NONCE}] In <repo>, add a comment line '// archie-e2e touch' to the top of README-adjacent file X and open a PR."
3. wait_for_task(nonce: NONCE), then resume with task_id + cursor while STATE=pending.
4. On STATE=approval_requested: read the approval type from the APPROVAL_TYPE= line of the wait_for_task output, then call approve(task_id: <id>, type: "edit_mode", approve: true) (this is the POST /api/tasks/:id/approve path).
5. Continue the wait_for_task loop; assert the task reaches STATE=completed.
6. Excerpts for evidence: events must show approval:requested (data.approvalType: "edit_mode") followed by approval:resolved and eventually task:completed; the knowledge log records the approval decision line.

Recipe: merge-approval-deny

An explicit merge request in a repo without autoMerge: true trips the merge approval gate, is denied via the API path, and no merge ever executes — the deny path keeps QA from really merging anything (pr-merge-policy AC3).

Prerequisite: same as edit-mode-approval — the workdir has at least one configured engineering repo, and that repo must not resolve to autoMerge: true (the shipped default: any repo whose declaring agents don't all set the flag). If the prerequisite is unmet, do NOT fake a pass — report the scenario as BLOCKED, stating: what is missing (no configured engineering repo in the workdir, or only auto-merge repos), what was attempted, and the remedy (configure a repo via the plugins setup, reboot, re-run). A BLOCKED scenario produces no evidence file; it appears in the run report only.

1. Mint a fresh nonce as above.
2. create_task with a small, real change request against a configured repo that ends in an open PR, e.g. "[${NONCE}] In <repo>, add a comment line '// archie-e2e touch' to the top of README-adjacent file X and open a PR. Do not merge it."
3. wait_for_task(nonce: NONCE), then resume with task_id + cursor while STATE=pending. On STATE=approval_requested with APPROVAL_TYPE=edit_mode: approve(task_id: <id>, type: "edit_mode", approve: true) and keep waiting.
4. Wait until the task settles with the PR opened (STATE=completed, PM reply announcing the PR). Note the PR number and repo from the PM reply or the knowledge log — the approve call in step 7 needs exactly this identity.
5. send_message(task_id: <id>, message: "[${NONCE}] Please merge that PR.") — the PM delegates to the repo agent, whose merge_pull_request call posts the merge approval prompt instead of merging.
6. Resume the wait_for_task loop; assert STATE=approval_requested with APPROVAL_TYPE=merge. The knowledge log now carries the decision finding Merge approval requested for <github>#<pr_number> — this names the identity to resolve against.
7. approve(task_id: <id>, type: "merge", approve: false, github: "<github>", pr_number: <pr_number>) with the identity of the PR the scenario drove open — the API rejects merge-type resolutions that omit github/pr_number.
8. Continue the wait_for_task loop; assert the task settles (STATE=completed) and no merge occurred: the knowledge log contains the denial finding Merge denied by user — PR not merged, and neither the log nor the events contain a merged completion (PR … merged on user approval) for this PR.
9. Excerpts for evidence: events must show approval:requested (data.approvalType: "merge") followed by approval:resolved (data.approve: false) with no merge in between; knowledge-log lines for the merge-approval request and the denial finding.

3. Capture evidence

Each scenario produces a <scenario>.json (canonical) + <scenario>.md (rendered for reviewers) pair, written by the validated writer:

# assemble the payload, then:
cat payload.json | npx tsx tools/e2e/evidence.ts --out-dir <dir>
# or: npx tsx tools/e2e/evidence.ts --in payload.json --out-dir <dir>

Destination: --out-dir flag → E2E_EVIDENCE_DIR env → default ./e2e-evidence/ at the repo root. The default is gitignored local scratch; Forge Stage 4 passes --out-dir openspec/changes/<change>/qa-evidence/, which is committed. One out-dir with ac_ids routing inside the files is the default convention; per-AC directories (qa-evidence/<AC-id>/) also work by pointing --out-dir per invocation.

All-or-nothing semantics: stdin is read fully before parsing; truncated or malformed JSON produces a classed error ("truncated JSON input from stdin" / "invalid JSON from stdin: …"), a non-zero exit, and no files. An invalid payload exits non-zero naming every validation error and writes nothing. The pair is written atomically (temp-file + rename) and transactionally — both files land or neither. A half-written evidence file is silent poison for the reviewer judging pass/fail from the file alone; the writer makes that state unrepresentable.

Schema: archie-e2e-evidence/v1 (field by field)

Field	Type	Meaning
schema	"archie-e2e-evidence/v1"	Schema tag, exact string.
scenario	string, kebab-case	Canonical recipe name (basic-nonce, edit-mode-approval, merge-approval-deny); names the output files.
ac_ids	string[], non-empty	Brief AC ids this scenario verifies, e.g. ["AC2"].
started_at / finished_at	ISO 8601 strings	Scenario wall-clock bounds.
environment.base_url	string	Resolved base URL the scenario ran against.
environment.git_branch	string	Branch under test (git branch --show-current).
environment.git_commit	string	Commit under test (git rev-parse --short HEAD).
nonce	string	The minted nonce, e.g. E2E-a1b2c3d4.
task_id	string	Correlated task id from wait_for_task.
terminal_state	enum	completed | stopped | approval_requested | pending | not_found.
assertions	array, non-empty	Each: id, description, expected, observed (all non-empty strings), pass (boolean).
excerpts.knowledge_log	string[]	Verbatim knowledge-log lines the assertions rest on.
excerpts.events	array	Verbatim event objects (from get_events) the assertions rest on.
result	"pass" | "fail"	Overall verdict; the validator enforces it equals the AND of assertion passes.

The markdown companion is rendered mechanically from the JSON in the same invocation (metadata header, assertion table, fenced excerpts, verdict); the JSON is canonical if they ever seem to diverge.

4. Teardown

npx tsx tools/e2e/teardown.ts

Runs docker compose down, then verifies via docker compose ps --all --format json that no containers remain for the project — survivors (including stopped-but-not-removed ones) are listed by name and the command exits non-zero. On success it prints a confirmation line suitable for pasting into evidence: Teardown clean: docker compose ps --all reports no containers for this project. Never end a scenario session without a clean teardown exit.