agent-host-e2e-tests

name

agent-host-e2e-tests

description

Use when writing, recording, updating, or troubleshooting the agent host end-to-end tests under src/vs/platform/agentHost/test/node/e2e (black-box tests that drive the whole agent host over the AHP protocol, using a CapiReplayProxy record/replay system for Claude/Copilot/Codex). Covers adding a cross-provider test, re-recording fixtures after an SDK bump, gating non-deterministic or platform-specific tests, and diagnosing replay cache misses.

Agent host end-to-end tests

These tests run the whole agent host end-to-end (real server, real bundled provider SDK/CLI, real AHP protocol) while replaying recorded model traffic from committed YAML fixtures — deterministic and tokenless.

Before doing anything, read the architecture + troubleshooting reference: src/vs/platform/agentHost/test/node/e2e/README.md

It documents the mental model, the fixture format, every config flag, and a symptom→cause→fix troubleshooting table. This skill is only the workflows; the README is the source of truth for how it works.

Non-negotiable rules

1. Replay is default and strict. No env var → serves committed fixtures, no token, no network. An unrecorded request is a hard cache miss that fails the run.
2. A model-backed fixture's filename is derived from the test title (${provider}-${slug}.yaml). Renaming such a test orphans its fixture — re-record after any rename. Tests explicitly registered with hostOnlyTest(...) share captures/empty.yaml.
3. Recording needs a real token (GITHUB_TOKEN or gh auth token) and talks to real CAPI. Only run it intentionally, with trivial/read-only prompts in temp dirs.
4. Never hand-write or hand-edit fixture contents (especially not secrets/paths). Fixtures are always produced by recording; normalization/redaction is the proxy's job.
5. Gate, don't fight. If a behavior can't replay deterministically, gate the test (see Workflow C) instead of loosening timeouts or the strict check.
6. Track every disabled variant. Keep e2e/KNOWN_ISSUES.md current with the test title, scope, expected and observed behavior, and a focused reproduction command. For suspected product bugs, begin with a self-contained explanation in complete sentences of what the user is trying to do, what fails, and the likely user impact; define feature-specific terms instead of relying on test names or implementation details. Record symptoms, not speculative root causes.

Workflow A — Add a cross-provider test

1. Add a test to the closest module under e2e/suites/, or create and register a focused suite module when the behavior is distinct. Use hostOnlyTest(context, ...) when crossing the model boundary would be a bug; otherwise use test(...). Drive turns with dispatchTurn(...) + context.client.waitForNotification(...); assert on AHP notifications, never on wall-clock timing.
2. Keep the prompt minimal and deterministic (fewer model turns → smaller, more robust fixtures).
3. Record fixtures for every enabled provider (Workflow B). Host-only tests need no per-test recording: the shared empty fixture remains strict and fails on any model request.
4. Review the diff (Workflow B step 3), then run the test in plain replay mode to confirm it's green, then commit the test + fixtures together.

Provider-specific assertions go in that provider's *.integrationTest.ts after the defineAgentHostE2ETests(config) call.

Workflow B — Record / re-record fixtures

Re-record when you add a test, or when a bundled SDK/CLI bump changes its wire behavior (new endpoint, different turn count, changed tool schema).

1. Ensure a token is available: gh auth token (or export GITHUB_TOKEN).
2. Record per provider:

   AGENT_HOST_REPLAY_RECORD=1 ./scripts/test-integration.sh --run \
     src/vs/platform/agentHost/test/node/e2e/providers/claudeAgentHostE2E.integrationTest.ts
   

   Repeat for copilotAgentHostE2E / codexAgentHostE2E as needed.
3. Review git diff on the fixtures: no local usernames/absolute paths, no tokens, no unreleased model ids. If something leaked, the fix is to extend normalization/redaction in capiReplayProxy.ts (_normalize + the *_RE redactors) and re-record — not to edit the fixture.
4. Run plain replay (no env var) to confirm green, then commit.

If an SDK now hits a new ancillary/bootstrap endpoint (a probe, not a real model turn), add it to capiStubs.ts (served, not recorded) instead of recording it — see how /models/session is handled.

Workflow C — When a test can't replay deterministically

Real-time streaming, mid-turn aborts, and POSIX-specific local execution (shell tools, pwd, git worktrees) don't replay reliably. Gate them precisely so you keep coverage where it works:

• Record-only (no deterministic replay at all): (RECORD ? test : test.skip)('…') — see can abort a running turn.
• Subagent fixtures stale after an SDK bump: re-record them (AGENT_HOST_REPLAY_RECORD=1 …). Subagent flows are the most SDK-version-sensitive (parent + child share one /v1/messages sequence), but replay reliably once re-recorded, so no gating is needed.
• POSIX-only (fails on Windows): gate with !isWindows, or a targeted per-provider flag when only one provider diverges. See the worktree and subagent-reopen tests.
• Provider/OS-specific replay: add a targeted config gate that still permits recording and unaffected platforms. See the Codex shell-tool Linux gate.

Always add a comment explaining why the gate exists. Also add or update the corresponding entry in e2e/KNOWN_ISSUES.md. When the variant is enabled again, remove or update the entry in the same change.

Verifying & troubleshooting

• Run a single provider in replay: ./scripts/test-integration.sh --run <path> (no env var).
• Filter to one test: add --grep "<test title fragment>".
• On a hang / timeout, read the runtime log first. For the Copilot provider, a failed test tails the most recent Copilot runtime (@github/copilot CLI) process-*.log into the test output ([agent-host-e2e] # … lines) — the SDK/CLI's own account of startup, auth, the model request, and the turn lifecycle. It runs at --log trace. A turn that never produced a model response, a panic, or an out-of-order/protocol error points at the SDK/CLI (re-record if a bump left the fixture stale; otherwise it's a real regression). Claude/Codex use their own runtimes and are not captured here. See the README's "A turn hangs or times out with no OS pattern".
• For any failure (cache miss, missing fixture, per-OS timeout, leaked PII, subagent staleness, accidental real-CAPI contact), go to the Troubleshooting section of the README — it maps each symptom to its cause and fix.