| name | agent-host-e2e-tests |
| description | Use when writing, recording, updating, or troubleshooting the agent host end-to-end tests under src/vs/platform/agentHost/test/node/protocol (black-box tests that drive the whole agent host over the AHP protocol, using a CapiReplayProxy record/replay system for Claude/Copilot/Codex). Covers adding a cross-provider test, re-recording fixtures after an SDK bump, gating non-deterministic or platform-specific tests, and diagnosing replay cache misses. |
Agent host end-to-end tests
These tests run the whole agent host end-to-end (real server, real bundled provider SDK/CLI, real AHP protocol) while replaying recorded model traffic from committed YAML fixtures — deterministic and tokenless.
Before doing anything, read the architecture + troubleshooting reference:
src/vs/platform/agentHost/test/node/protocol/README.md
It documents the mental model, the fixture format, every config flag, and a symptom→cause→fix troubleshooting table. This skill is only the workflows; the README is the source of truth for how it works.
Non-negotiable rules
- Replay is default and strict. No env var → serves committed fixtures, no token, no network. An unrecorded request is a hard cache miss that fails the run.
- A fixture's filename is derived from the test title (
${provider}-${slug}.yaml). Renaming a test orphans its fixture — re-record after any rename.
- Recording needs a real token (
GITHUB_TOKEN or gh auth token) and talks to real CAPI. Only run it intentionally, with trivial/read-only prompts in temp dirs.
- Never hand-write or hand-edit fixture contents (especially not secrets/paths). Fixtures are always produced by recording; normalization/redaction is the proxy's job.
- Gate, don't fight. If a behavior can't replay deterministically, gate the test (see Workflow C) instead of loosening timeouts or the strict check.
Workflow A — Add a cross-provider test
- Add a
test(...) inside defineAgentHostE2ETests in agentHostE2ETestHelpers.ts. Drive it with dispatchTurn(...) + client.waitForNotification(...); assert on AHP notifications, never on wall-clock timing.
- Keep the prompt minimal and deterministic (fewer model turns → smaller, more robust fixtures).
- Record fixtures for every enabled provider (Workflow B).
- Review the diff (Workflow B step 3), then run the test in plain replay mode to confirm it's green, then commit the test + fixtures together.
Provider-specific assertions go in that provider's *.integrationTest.ts after the defineAgentHostE2ETests(config) call.
Workflow B — Record / re-record fixtures
Re-record when you add a test, or when a bundled SDK/CLI bump changes its wire behavior (new endpoint, different turn count, changed tool schema).
- Ensure a token is available:
gh auth token (or export GITHUB_TOKEN).
- Record per provider:
AGENT_HOST_REPLAY_RECORD=1 ./scripts/test-integration.sh --run \
src/vs/platform/agentHost/test/node/protocol/claudeAgentHostE2E.integrationTest.ts
Repeat for copilotAgentHostE2E / codexAgentHostE2E as needed.
- Review
git diff on the fixtures: no local usernames/absolute paths, no tokens, no unreleased model ids. If something leaked, the fix is to extend normalization/redaction in capiReplayProxy.ts (_normalize + the *_RE redactors) and re-record — not to edit the fixture.
- Run plain replay (no env var) to confirm green, then commit.
If an SDK now hits a new ancillary/bootstrap endpoint (a probe, not a real model turn), add it to capiStubs.ts (served, not recorded) instead of recording it — see how /models/session is handled.
Workflow C — When a test can't replay deterministically
Real-time streaming, mid-turn aborts, and POSIX-specific local execution (shell tools, pwd, git worktrees) don't replay reliably. Gate them precisely so you keep coverage where it works:
- Record-only (no deterministic replay at all):
(RECORD ? test : test.skip)('…') — see can abort a running turn.
- Subagent fixtures stale after an SDK bump: re-record them (
AGENT_HOST_REPLAY_RECORD=1 …). Subagent flows are the most SDK-version-sensitive (parent + child share one /v1/messages sequence), but replay reliably once re-recorded, so no gating is needed.
- POSIX-only (fails on Windows): gate with
!isWindows, or a per-provider flag like shellPermissionReplayUnstableOnWindows when only one provider diverges. See the worktree and shell-permission tests.
Always add a comment explaining why the gate exists.
Verifying & troubleshooting
- Run a single provider in replay:
./scripts/test-integration.sh --run <path> (no env var).
- Filter to one test: add
--grep "<test title fragment>".
- For any failure (
cache miss, missing fixture, per-OS timeout, leaked PII, subagent staleness, accidental real-CAPI contact), go to the Troubleshooting section of the README — it maps each symptom to its cause and fix.
