writing-cli-e2e-tests

# Writing CLI E2E Tests ## Overview E2e tests run real CLI commands against real infrastructure with real side effects. They validate that commands work end-to-end for both humans (interactive) and agents/CI (non-interactive). This skill defines the philosophy and patterns for writing effective e2e tests in `packages/@sanity/cli-e2e/`. For local setup, CI behavior, and how to add new env vars, see [`packages/@sanity/cli-e2e/README.md`](../../../packages/@sanity/cli-e2e/README.md). ## When to Use - Adding e2e test coverage for a new CLI command - Adding tests for new flags or flows on an existing command - Modifying CLI behavior that needs e2e validation ## Discovery & Planning Before writing any tests, read the command source code and present a plan for user approval. **Step 1: Understand the command.** Run the command interactively and with `--help` to observe its behavior. Identify: - All flags and arguments the command accepts - All distinct flows and code paths (e.g., `init` has studio, app, and Next.js paths) - Which prompts appear and in what order - What side effects the command produces (files, API calls, data) Discover behavior by running the CLI, not by reading source code. The test should verify what the user experiences, and the actual prompt flow may differ from what the source suggests. **Step 2: Plan the test structure.** Present to the user: - Proposed file structure (`__tests__/<command>/...`) - For each file, the list of test names with the CLI command each will run - Which tests are non-interactive vs interactive - Any inherently incomplete flows (abort, unauthenticated) **Step 3: Get user approval.** Wait for the user to review and revise the plan before writing any test code. Example plan output (describe behaviors, not final `test()` names — those get refined during implementation): ``` __tests__/init/ init.test.ts - smoke: rejects a known-bad flag (no auth) - --bare mode prints project info without creating any files (non-interactive) init.studio.test.ts describe.each([{-y}, {no -y}]) — non-interactive, both unattended modes - default flow creates a TypeScript studio with the expected config files - --no-typescript produces JavaScript files init.studio-interactive.test.ts - Ctrl+C at the first prompt aborts with exit code 130 - walks through all prompts (template, TypeScript, package manager) and produces a working studio init.app.test.ts describe.each([{-y}, {no -y}]) — non-interactive - app-quickstart template produces the expected app scaffold - interactive: project/dataset prompts appear and selectable ``` ## Core Principles ### 1. Test Complete Flows With Thorough Assertions Run commands to their final outcome — files created, data written, success message printed. A half-finished flow that kills the session partway gives false confidence. Since each full run is expensive, assert everything meaningful from that single invocation: exit code, stdout messages, generated files, config content. ### 2. Cover Both Execution Modes Every command should have both interactive and non-interactive tests. Non-interactive tests verify agents and CI automation can use the command fully. Interactive tests verify humans get proper prompts, navigation, and feedback. Both modes must work completely — they serve different audiences. ### Inherently Incomplete Flows Some tests cannot run to completion by design. These are the only acceptable exceptions to the "complete flows" principle: - **Abort handling (Ctrl+C):** One test per command that sends `sendControl('c')` at the earliest prompt and awaits `session.waitForExit(130)` (SIGINT). One abort test is sufficient — the abort mechanism is the same regardless of which prompt stage it fires at, so testing Ctrl+C at multiple stages adds cost without value. - **Unauthenticated prompts:** Tests that verify login prompts appear when no token is provided. These can't complete because there's no real login flow in tests. Keep these minimal — one of each per command, at the top level (no `describe` wrapper). Every other test should run to completion. ## Choosing Execution Mode **Is the behavior driven by user prompts/navigation?** - **YES** → Does the command support flags that bypass the prompts? - **YES** → Write BOTH: non-interactive test with flags AND interactive test with prompts - **NO** → Interactive test only (PTY) - **NO** → Non-interactive test only (spawn) **Non-interactive (spawn):** Command fully driven by flags/args. Fast, reliable, easy stdout/stderr assertions. **Interactive (PTY):** Tests the prompt experience — selection navigation, text input, abort handling. **Both:** Most commands should have both. Non-interactive proves the command works. Interactive proves the UX works. Apply this per-flow, not per-command — if a command has studio, app, and Next.js flows, each flow needs both modes. ## Test File Organization All commands get a folder. Split non-interactive and interactive tests into separate files (e.g., `init.studio.test.ts` and `init.studio-interactive.test.ts`) — this improves vitest sharding and keeps files focused. **For commands with multiple distinct flows**, split files by user flow or functionality: ``` __tests__/ init/ init.studio.test.ts init.nextjs.test.ts init.bare.test.ts init.errors.test.ts deploy/ deploy.app.test.ts deploy.studio.test.ts ``` **For simpler commands**, a single test file is enough: ``` __tests__/ datasets/ datasets.test.ts help/ help.test.ts ``` **Don't create single-test files.** If a flow only has 1-2 tests (e.g., a smoke test or a simple mode like `--bare`), merge it into the command's base `<command>.test.ts` file rather than giving it a dedicated file. **Test naming:** Descriptive names that accurately describe the behavior being verified. No numeric IDs. The name must match what the test actually asserts — a test for a deprecated flag should say "deprecated", not "invalid input". ```typescript // GOOD test('creates TypeScript studio with correct config files', ...) test('rejects deprecated --reconfigure flag', ...) // BAD test('2.4 default init creates correct TypeScript project', ...) test('rejects invalid input with helpful error', ...) // if it's testing a deprecated flag, not invalid input ``` ## Test Structure Style **Flat over nested.** Don't wrap single tests in `describe` blocks. Only use `describe` when tests share setup/teardown or parameterization (`describe.each`). Don't use `describe` just to label categories like "abort handling" or "complete flows" — a flat list under one top-level `describe` is clearer. **`beforeEach`/`afterEach` for cleanup.** Use lifecycle hooks for temp directory creation and cleanup instead of `try/finally` in every test: ```typescript describe('sanity init', {timeout: 120_000}, () => { let tmp: Awaited<ReturnType<typeof createTmpDir>> beforeEach(async () => { tmp = await createTmpDir({useSystemTmp: true}) }) afterEach(async () => { await tmp.cleanup() }) test('creates studio', async () => { // use tmp.path directly — no try/finally needed }) }) ``` **`test.each` for variants.** When testing the same flow with different inputs (e.g., templates, package managers), use `test.each` instead of duplicating tests: ```typescript test.each(['clean', 'blog'])('creates studio with %s template', async (template) => { // ... }) ``` **Consolidate interactive prompt tests.** When a command has multiple sequential prompts (template, TypeScript, package manager), write one test that walks through all of them rather than separate tests that each omit one flag. Each CLI invocation is expensive (~12s); one test that exercises three prompts is better than three tests that each exercise one. ```typescript // BAD: three separate tests, three CLI invocations test('shows template selection', ...) // omits --template test('shows TypeScript prompt', ...) // omits --typescript test('shows package manager prompt', ...) // omits --package-manager // GOOD: one test, one invocation, all prompts exercised test('walks through template, TypeScript, and package manager prompts', async () => { const session = await runCli({ args: ['init', '--project', projectId, '--dataset', 'production', '--output-path', tmp.path, '--no-mcp', '--no-git'], interactive: true, }) await session.waitForText(/Select project template/i) session.sendKey('Enter') await session.waitForText(/Do you want to use TypeScript/i) session.sendKey('Enter') await session.waitForText(/package manager/i) session.sendKey('Enter') await session.waitForExit(0, 90_000) expect(existsSync(`${tmp.path}/sanity.config.ts`)).toBe(true) }) ``` **Minimal flags.** For non-interactive tests, only include flags the test is specifically testing. For interactive tests, leave prompts unpinned so `selectOption` exercises them — only pin infrastructure values like `--organization` or `--output-path` that aren't UX under test. Use `--no-git` and `--no-mcp` to skip side effects irrelevant to the flow being tested. **Precise assertions.** Assert on actual content, not proxies: ```typescript // BAD: vague, tells you nothing on failure expect(stderr.length).toBeGreaterThan(0) expect(stdout).toMatch(/import/i) expect(exitCode).not.toBe(0) // GOOD: specific, failure message is immediately useful expect(stderr).toContain('--reconfigure is deprecated') expect(stdout).toMatch(/Done! Imported \d+ documents/) expect(exitCode).toBe(1) ``` **Inline args, don't abstract.** Inline CLI args directly in each test rather than building helper functions that produce them — abstraction obscures what each test actually runs. The full `runCli()` API and interactive session methods are documented in [`packages/@sanity/cli-e2e/README.md`](../../../packages/@sanity/cli-e2e/README.md); shared `testFixture()` / `createTmpDir()` helpers live in `@sanity/cli-test`. ## Test Structure Patterns ### Non-Interactive Complete Flow ```typescript describe('sanity init - studio', {timeout: 120_000}, () => { let tmp: Awaited<ReturnType<typeof createTmpDir>> beforeEach(async () => { tmp = await createTmpDir({useSystemTmp: true}) }) afterEach(async () => { await tmp.cleanup() }) test('creates studio with TypeScript and correct config', async () => { const {error, stdout} = await runCli({ args: ['init', '-y', '--project', projectId, '--dataset', 'production', '--output-path', tmp.path, '--typescript'], }) if (error) throw error expect(existsSync(`${tmp.path}/sanity.config.ts`)).toBe(true) expect(existsSync(`${tmp.path}/package.json`)).toBe(true) const config = readFileSync(`${tmp.path}/sanity.cli.ts`, 'utf8') expect(config).toContain(projectId) expect(stdout).toMatch(/sanity docs|sanity help/i) }) }) ``` ### Interactive Complete Flow Use `selectOption(pattern)` to navigate select prompts by text instead of counting ArrowDown presses. It scrolls through the list, handles off-screen options, and throws if zero or multiple options match. Pass a string for a literal substring match or a regex for pattern matching (e.g., matching a project ID inside `"Project Name (id)"`). Strings are escaped before matching, so they're treated as literal text — but they still match anywhere in the option line, so prefer regex with anchors when collisions are likely (e.g., `'dev'` would also match `'development'`). ```typescript test('complete interactive flow selects project and dataset', async () => { const session = await runCli({ args: ['init', '--template', 'app-quickstart', '--organization', orgId, '--output-path', tmp.path, '--no-git', '--no-mcp'], interactive: true, }) await session.waitForText(/Configure a project for this app/i) await session.selectOption(new RegExp(`\\(${projectId}\\)`)) await session.waitForText(/Select dataset to use/i) await session.selectOption('production') await session.waitForText(/Package manager to use/i) await session.selectOption('pnpm') await session.waitForExit(0, 90_000) expect(existsSync(`${tmp.path}/src/App.tsx`)).toBe(true) }) ``` ### Testing Unauthenticated Flows Auth token is always available. To test unauthenticated behavior, override the env: ```typescript test('no token triggers login prompt', async () => { const session = await runCli({ args: ['init'], env: {SANITY_AUTH_TOKEN: ''}, interactive: true, }) await session.waitForText(/log in|create.*account/i) // ... complete the flow }) ``` ## Testing Features Not Yet on `latest` The scheduled workflow runs the suite against `sanity@latest` from npm, not the working tree (see the README's CI section). A test for a feature that hasn't shipped to the `latest` dist-tag yet passes on PRs (working-tree CLI) but breaks the hourly scheduled run — it hits a published binary that doesn't have the feature. Guard those tests so they only run against a CLI that has the feature: ```typescript const isRegistryMode = process.env.E2E_REGISTRY_MODE === 'true' describe.skipIf(isRegistryMode)('sanity dev (workbench)', () => { // ... }) ``` Remove the guard once the feature lands on `latest`. ## Timeouts Set timeouts at the `describe` block level. The vitest config provides defaults (`testTimeout: 30_000`). If a group of tests needs more time, set it once: ```typescript describe('studio creation flows', {timeout: 120_000}, () => { test('creates studio with TypeScript', async () => { // inherits describe timeout }) test('creates studio with JavaScript', async () => { // inherits describe timeout }) }) ``` ## Interactive Test Resilience Interactive tests must not depend on list order or item positions. Use `selectOption` to find items by text — it handles scrolling and position automatically. ```typescript // BAD: counts ArrowDown presses, breaks if list order changes session.sendKey('ArrowDown') session.sendKey('ArrowDown') session.sendKey('Enter') // GOOD: finds the option by text regardless of position await session.selectOption('production') await session.selectOption(new RegExp(`\\(${projectId}\\)`)) ``` For assertions on output, match structural prompt text, not dynamic API content: ```typescript // BAD: asserts on specific dynamic content expect(output).toContain('My Specific Project Name') // GOOD: asserts on prompt structure or known output strings await session.waitForText(/Select project|Configure a project/i) expect(output).toContain('Your custom app has been scaffolded') ```