| name | goose-gui-e2e |
| description | Drive the real Goose desktop GUI (Electron) as an MCP client via Playwright to end-to-end test this template's MCP Apps - asserting the ui:// iframe actually renders and round-trips in a real host. Use to run, add, or debug MCP-App e2e scenarios. |
Goose-GUI end-to-end testing of the MCP Apps
Goose (Block's open-source agent, an Agentic AI Foundation reference MCP client)
renders MCP Apps natively - it fetches a tool's ui:// resource and mounts the
HTML in a sandboxed iframe. This skill stands up the real Goose desktop app
(Electron), wires this template's /mcp mount into it as a streamable-HTTP
extension, drives a chat scenario with Playwright-Electron (DOM selectors, no
pixel coordinates), and decides PASS/FAIL from two records the mock LLM cannot
fake: the rendered iframe DOM and the tool-call round-trip log.
This closes the L4 "rendered iframe in a real host" gap from issue #37 - the layer
tests/test_mcp_e2e.py (wire) and the per-app vitest (component) deliberately
don't cover. It's ported from Edison-Watch's goose-gui-e2e harness; the Goose
build under $E2E_HOME/goose_src is repo-agnostic and shared with it.
scenario (JSON: prompt + tool plan + expect)
│ Playwright-Electron drives the DOM
▼
mock LLM ──emits webhook_settings──► Goose desktop GUI ──/mcp──► template server
(scenario engine) (Electron, real) (FastMCP + enhancer
▲ │ renders ui:// attaches ui://mymcp/settings)
│ ▼
└────────────────────────── sandboxed iframe (Settings app)
│
assert: iframe DOM (pw_result) + tool round-trip (toolcalls.jsonl)
Quickstart
cd .agents/skills/goose-gui-e2e/scripts
export E2E_HOME="$HOME/goose-e2e"
bash setup.sh
bash up.sh
bash run_test.sh settings_render
bash run_all.sh
bash down.sh
make test_apps_e2e runs up.sh && run_all.sh. Each run_test.sh writes a
screenshot of the real Goose GUI (with the Settings app rendered) to
$E2E_HOME/shot_<scenario>.png.
Where this runs in the suite
Heavy tier - not in make ci (which stays Node/Rust/Electron-free):
- Local / make:
make test_apps_e2e (after the one-time setup.sh).
- Full pytest suite:
tests/test_apps_e2e.py is a collectable member, skipped
by default; opt in with RUN_APPS_E2E=1 (e.g. RUN_APPS_E2E=1 uv run pytest tests/test_apps_e2e.py --no-cov). It shells out to up.sh + run_all.sh.
(--no-cov keeps a single-file run from tripping pytest.ini's coverage gate.)
- CI:
.github/workflows/apps_e2e.yaml - workflow_dispatch + weekly cron. It
builds Goose via setup.sh (open egress, so the npmmirror preconditions below
don't apply on GitHub runners) and runs the guarded pytest entry.
Preconditions (verify before setup)
- npmmirror on the network allowlist. Electron's binary is fetched from a
mirror because GitHub release assets are egress-blocked in the sandbox. In the
environment's Network access → Custom, add
registry.npmmirror.com and
cdn.npmmirror.com (keep the default package-manager list checked). setup.sh
preflights this and fails loudly if it's missing.
- Sandbox provides:
cargo, pnpm/node, bun, uv, xvfb-run. setup.sh
uses git clone (allowed) + the mirror - none of the GitHub-blocked paths.
How a scenario works
A scenario is JSON in scripts/scenarios/<name>.json:
{
"prompt": "Open my settings.",
"plan": [ { "match": "webhook_settings" } ],
"final": "Opened your settings.",
"app_uri": "ui://mymcp/settings",
"expect": {
"tool_called": "webhook_settings",
"round_trip": true,
"app_rendered": true,
"dom_contains": ["Settings", "Gmail"]
}
}
prompt - the user message typed into Goose.plan - the tool-call sequence the mock emits. match is a substring of
the tool name Goose offers (Goose may namespace extension tools, so substring
not exact). A plan step may carry "args" (e.g. {"match": "gmail_get_thread", "args": {"thread_id": "t-1001"}}) which the mock passes through as the tool
call's arguments. Pick an app-returning tool: webhook_settings →
ui://mymcp/settings renders offline with zero external deps, and
gmail_get_thread → ui://mymcp/gmail_inbox renders offline because the
stack sets GMAIL_FAKE_BACKEND=1 (see below).expect - checked by mcp_probe.py: tool_called + round_trip (from the
tool-call log) and app_rendered + dom_contains (from the rendered iframe).
Rendering the Gmail apps offline (GMAIL_FAKE_BACKEND)
The gmail_inbox / gmail_composer apps normally need a linked Google account:
gmail_get_thread → _get_gmail_client() raises GmailNotConnectedError
(ConnectRequiredError) with no token row, so the tool errors before
send_app and the iframe never mounts. To render them offline, up.sh starts
the server with GMAIL_FAKE_BACKEND=1, which makes _get_gmail_client return a
fixture-serving fake (services/_gmail_fake_backend.py) instead of hitting
Google - no linked account, OAuth, or network. The fixtures are raw-Gmail-API
shaped, so the real service parsing → Pydantic GmailThread runs end-to-end;
only the network boundary is faked. The flag is hard-refused under
DEV_ENV=prod (guard in services.gmail_svc._maybe_fake_gmail_client), so it
can never stand in for a real mailbox in production.
Why the fake also serves the curated inbox: the gmail_inbox app registers
its ontoolresult handler in a mount effect, and if the host delivers the thread
result before that runs, the app misses it and falls back to a curated-inbox
refresh (gmail_inbox.refresh → gmail_curate_inbox) after ~800ms. So a
thread-open renders the reader when it wins that race and the curated list
when it loses. The fake serves both paths from the same fixture thread, and
gmail_thread_render asserts on content present in both views, so the scenario is
deterministic instead of ~30% flaky. gmail_get_thread's round-trip is still
proven independently via the tool-call log regardless of which view renders.
Optional: a click → callServerTool → re-render step
A scenario may add an interact block to drive a real control inside the app
iframe after it renders, exercising a user-initiated server round-trip (the
settings_subscribe fixture does this - clicks "Add endpoint" → settings.subscribe):
"expect": { "...": "...", "interaction_rendered": true },
"interact": {
"fill": { "selector": "input[type=url]", "value": "https://webhook.example.com/e2e" },
"click": { "text": "Add endpoint" },
"expect_dom_contains": ["Signing secret", "webhook.example.com"]
}
The iframe's callServerTool goes app → Goose → /mcp directly, bypassing the
mock LLM, so this round-trip never appears in the tool-call log - the re-rendered
DOM (expect_dom_contains, here the returned signing secret) is the only proof,
and only the server's real response can produce it. For post-interaction DOM that
plain text can't see, interact also accepts "expect_selectors": ["css", ...] -
CSS selectors that must each match at least one element (e.g.
img[src^="data:image/png;base64,"] to prove a fetched image's data URI was
written into the iframe DOM - presence, not paint). interaction_rendered: true
makes mcp_probe.py require it, and pw_scenario.mjs fails the drive if the
post-click DOM never matches. This needs the "Add endpoint" control, which renders
only when push_available is true - up.sh sets GMAIL_PUBSUB_TOPIC to ungate it
(no Pub/Sub is ever contacted).
Why the mock cannot fake a PASS
The mock only follows the plan and returns canned narration; it never inspects
tool results. The verdict is computed by mcp_probe.py from two records the mock
doesn't control:
- the rendered iframe -
pw_scenario.mjs frameLocators into the real Goose
DOM and asserts the Settings app's text. The iframe only exists if the server
returned the app resource and Goose fetched + mounted it - a full round-trip.
- the tool-call log - the calls Goose actually issued to
/mcp and the
results the server returned.
Two gates stop a no-op run from passing on stale data:
- Drive must render.
pw_scenario.mjs exits non-zero unless the app iframe
rendered with the expected DOM; run_test.sh checks it (PIPESTATUS[0]).
- Log must be new.
run_test.sh snapshots the tool-call-log line count before
driving; mcp_probe.py FAILs if no new entries were appended for this run.
Mock vs. real LLM
- Mock (default): deterministic, offline, CI-friendly.
configure_goose.sh mock.
- Real LLM:
configure_goose.sh real anthropic $KEY (model via $GOOSE_REAL_MODEL).
The agent decides tool calls from the prompt; requires an API key and the
provider host on the allowlist. The render/round-trip assertions are unchanged.
Sandbox gotchas already handled (do not re-discover)
Baked into the scripts; listed so you recognize them if something drifts:
| Symptom | Cause → handled by |
|---|
| Electron binary 403 | GitHub release assets blocked → mirror + checksum (setup.sh) |
@electron/node-gyp 403 on pnpm install | git tarball on codeload blocked → registry override (setup.sh) |
| Electron SIGSEGV "Missing X server" | needs a display → xvfb-run owns Xvfb per test |
| Electron crash under Xvfb | --no-sandbox --disable-gpu (NOT --in-process-gpu) |
CDP /json empty / debug-port hang | Playwright-Electron launches over a CDP pipe |
| bg service dies next tool call | setsid ... & disown (start_detached); re-run up.sh to recover |
/mcp 401 | the extension must send the seeded X-API-KEY header (configure_goose.sh) |
| app iframe not found | assert only on an app-returning tool; the Settings app renders offline |
Files
| File | Role |
|---|
lib.sh | paths/ports + bounded-wait/setsid helpers |
setup.sh | one-time provisioning: build goose, Electron via mirror, pnpm install, dev bundle |
seed.py | create the e2e SQLite schema + one API key; prints the raw key |
up.sh / down.sh | idempotent stack bring-up / teardown |
configure_goose.sh | write Goose config for mock/real, wiring the mymcp extension at /mcp |
mock_llm.py | scenario-engine mock LLM (follows the plan, logs tool calls) |
pw_scenario.mjs | Playwright-Electron driver + rendered-iframe DOM assertion |
mcp_probe.py | assertion oracle - rendered iframe (pw_result) + tool round-trip (toolcalls.jsonl) |
run_test.sh / run_all.sh | drive + assert one / all scenarios |
scenarios/settings_render.json | scenario 1: LLM opens the Settings app; assert it renders |
scenarios/settings_subscribe.json | scenario 2: click "Add endpoint" → settings.subscribe → assert the re-render |
scenarios/gmail_thread_render.json | scenario 3: LLM opens a thread via gmail_get_thread; assert the gmail_inbox reader iframe renders (needs GMAIL_FAKE_BACKEND) |
scenarios/gmail_remote_images.json | scenario 4: remote-image pipeline - assert the blocked-by-default "Show images" banner renders, click it, and assert the gmail_inbox.fetch_image round-trip re-renders as "Retry" (fixture URL is .invalid, so the SSRF guard rejects it deterministically offline and in CI) |
Also outside the skill: tests/test_apps_e2e.py (guarded pytest entry) and
.github/workflows/apps_e2e.yaml (opt-in CI).
