playwright-e2e

# Playwright E2E Tests Guidelines for writing and iterating on Playwright tests for VS Code extensions. ## Required Reading **Read ALL before responding:** - `references/coding-playwright-tests.md` - Writing tests - `references/local-setup.md` - Scratch org setup (Dreamhouse, minimal, non-tracking) - `references/iterating-playwright-tests.md` - Iterating on tests ("Things to ignore" for failure analysis) - `references/analyze-e2e.md` - Analyzing E2E test results from CI # Use playwright-vscode-ext Shared code (helpers, locators, configuration) for tests. **Desktop workspace shapes (pick one per test):** - **No folder open** — fixture opens a Salesforce project, then call `prepareNoFolderOpenForPaletteTests(page)` (runs `Workspaces: Close Workspace` + workbench wait). Or use `closeWorkspaceToEmptyWindow` if UI is already prepared. - **Folder open, no `sfdx-project.json`** — `createDesktopTest({ emptyWorkspace: true })`; workspace path comes from `createEmptyTestWorkspace()` (also exported from the package). - **Default org in workspace** — pass `orgAlias: '…'` (e.g. `MINIMAL_ORG_ALIAS` / `NON_TRACKING_ORG_ALIAS` / `DREAMHOUSE_ORG_ALIAS`) so `.sfdx/config.json` gets `target-org`. Omit `orgAlias` or use `undefined` for **no** `config.json` (no org). - **Multi-package directory, no org** — `multiPackageNoOrgDesktopTest` (extend `noOrgDesktopTest`); creates a temp workspace with `sfdx-project.json` listing multiple `packageDirectories` (`force-app`, `extra-pkg`). Use `multiPackageNoOrgTest` from `fixtures/index.ts` in test files. **VSIX mode** (`useVsix` option): - `createDesktopTest({ useVsix: true })` — installs built VSIXs into a hash-keyed cache dir (`.vscode-test/ext-<hash>/`) and launches VS Code with `--extensions-dir` instead of `--extensionDevelopmentPath`. Exercises real shipping artifact (bundled `dist/`, `.vscodeignore`, `packageUpdates`). - Installs requested local VSIX dirs in `extensionDependencies` order (from each local `package.json`), so local dependency VSIXs install before dependents. - Default: `process.env.E2E_FROM_VSIX === '1'` — set in CI to enable without code changes. - Requires `vscode:package` to have run first (produces `.vsix` in package dir). `test:desktop` depends on `vscode:package` for this reason. - Idempotent across parallel workers: atomic rename; second worker skips if cache exists. **Code Builder container mode** (`createContainerConfig`, `createContainerTest`): - `createContainerConfig({ testDir: '…' })` — config for driving tests against a running Code Builder container. Container lifecycle (run, extension swap, health checks) managed by orchestrator/CI, not Playwright. Tests drive a browser-client to the container URL (like web mode) while the workbench runs the desktop extension build. Use env `CODE_BUILDER_URL` (defaults to `http://localhost:8123`). - `createContainerTest()` — fixture that navigates a plain Chromium `page` to the container workbench and waits for readiness. Specs reuse existing page objects (commands, helpers, locators) unchanged. - Seeding: `seedWorkspace` (exported from the toolkit) handles post-boot writes (`coder.json` path + workspace-trust setting). Works with fixture projects mounted into the container (e.g. `test/playwright/fixtures/container-workspace/`). Call after workbench readiness, before extension swap/restart. ## Span files (when debugging traces) Available local + CI/GHA. - Output: `~/.sf/vscode-spans/` — `web-*.jsonl` (test:web), `node-*.jsonl` (test:desktop) - Auto-enabled (no manual enable needed) - CI runs: copied into package `test-results/spans/` artifacts (see workflow upload/download in `references/analyze-e2e.md`) - Latest: `ls -lt ~/.sf/vscode-spans/` - Clear before run for fresh output: `rm -rf ~/.sf/vscode-spans/` - Format: JSONL; parse each line with `JSON.parse` - Fields: `name`, `traceId`, `spanId`, `parentSpanId`, `durationMs`, `status`, `startTime`, `attributes` See `.claude/skills/span-file-export/SKILL.md` for enable/OTLP vs file. ## Telemetry inspection (diagnostic tests) Desktop tests can inspect both telemetry pipelines on-disk for diagnostic/integration testing: 1. **O11y spans** — produced by services Effect pipeline; written to `~/.sf/vscode-spans/*.jsonl` (auto-enabled) 2. **AppInsights events** — produced by class-based TelemetryFile reporter when `localTelemetryLogging` is enabled; written to `{workspace}/salesforcedx-vscode-core-telemetry.json` (AppInsights shape, real client inert in dev/test) **Fixture setup:** Enable both pipelines by passing `additionalExtensionDirs: ['salesforcedx-vscode-core']` (for core extension + TelemetryFile reporter) and `userSettings: { 'telemetry.telemetryLevel': 'all', 'salesforcedx-vscode-core.advanced.localTelemetryLogging': 'true' }` to `createDesktopTest`. Launch against a real scratch org (e.g., `orgAlias: MINIMAL_ORG_ALIAS`) so org-identity attributes populate in both pipelines. **Reading artifacts:** - Spans: parse `~/.sf/vscode-spans/*.jsonl` line-by-line with `JSON.parse` - AppInsights events: file contains comma-separated pretty JSON objects; wrap in `[]` and strip trailing comma to parse: `JSON.parse(`[${raw.trim().replace(/,\s*$/, '')}]`)` **Pattern:** Run command, capture artifacts, reload window to flush TelemetryFile buffer (fires deactivationEvent), then assert event/span presence + attributes. See `packages/salesforcedx-vscode-lightning/test/playwright/specs/telemetryOutput.desktop.spec.ts` for example. ## Checking for Scratch Orgs If you aren't sure if orgs are set up locally, ```bash sf org list ``` Look for the required org aliases (e.g., `minimalTestOrg`, `nonTrackingTestOrg`, `orgBrowserDreamhouseTestOrg`). If missing, create them using the appropriate setup commands from `references/local-setup.md`. **Pro tip**: Use `sf org list --json | jq '.result.scratchOrgs[] | select(.alias) | .alias'` to list only scratch org aliases. ## Running tests (AI behavior) When running Playwright tests (`npm run test:web`, `test:desktop`, etc.), never block >30s. Use `is_background: true` so tests run while the AI continues. Check terminal output or `output_file` later. ## Apex OAS E2E Tests Playwright desktop tests live in `packages/salesforcedx-vscode-apex-oas/test/playwright/specs/` with dedicated CI workflow `.github/workflows/apexOasE2E.yml` (macOS + ubuntu, desktop only). Specs share one MINIMAL_ORG_ALIAS scratch org and are serialized via `workers: 1` in `playwright.config.desktop.ts`. Tests that deploy ESR metadata requiring API >=66 call `setWorkspaceApiVersion()` to bump the fixture's default sourceApiVersion (64.0 → 66.0). Specs that click modal-dialog buttons require `window.dialogStyle: custom` in the fixture's `userSettings`. The OAS REST generation path requires an LLM service registered with the VS Code service provider — supplied at runtime by A4V (`salesforce.salesforcedx-einstein-gpt`), which is no longer a declared `extensionDependency`. The AuraEnabled path needs only an active org. Specs exercising REST generation install A4V via the desktop fixture's pre-launch step; obtaining the LLM service is fail-fast, so `waitForA4VAndOasCommands` calls `waitForExtensionsActivated` to ensure the provider has registered its command before generation runs. **A4V LLM rate limit = skip, not fail (pre-migration):** the shared Core model exhausting its monthly quota is an infra outage, not a product bug — it can hit *any* spec that triggers a generation LLM call (all composed/decomposed/context-menu specs), not just manual-merge. The extension surfaces it as a real error notification (`/monthly rate limit/`, from the `llm_monthly_rate_limit` i18n message) instead of the old generic "LLM did not return any content", so specs detect it straight from the UI — no span-file scan. Wrap the generation success signal with `assertGenerationOrSkipOnRateLimit(test, page, success)` (oasHelpers): `success` is the success assertion (`expect(tab).toBeVisible()` or `waitForEsrFile(...)`); it races the rate-limit notification and `test.skip`s if that wins, else resolves/rethrows. The eligibility-failure specs (`ineligibleClass`, `mixedFrameworksClass`, `restResourceNoHttpMethod`) fail before any LLM call and need no guard. ## Don't use the clipboard to set editor content `navigator.clipboard.writeText` + Paste is a shared global resource — desktop Electron clipboard is the system OS clipboard (electronjs.org/docs/latest/api/clipboard), so parallel workers (`fullyParallel: true`) race: worker B's write between A's write and A's Paste makes A paste B's text. Flaky, hard to diagnose. Set content directly instead: - type via `page.keyboard.type(text)` after `focusMonacoInput` + Select All + Delete ([editor selection](references/coding-playwright-tests.md#commands-with-editor-selection)), or - write the file on disk (desktop fs / web memfs), or - set the editor model value through a VS Code command. Note: "keyboard shortcut can miss on web" comments refer to **shortcut keystrokes** (`Cmd+A`/`Cmd+V`) not landing — fix is command-palette `Select All`/`Paste`, not clipboard. Clipboard ≠ required for that. ## Running Full E2E Test Suite See `references/full-suite-execution.md` for complete guide on running all E2E tests locally across all 9 packages in correct dependency order with failure analysis. ## Disable/reenable other E2E when iterating To run only your new test in CI while iterating: 1. **Disable other workflows** — add your branch to `branches-ignore` in `.github/workflows/*.yml` that have `push: branches-ignore: [main, develop]` (e.g. `testCommitExceptMain.yml`, `coreE2E.yml`, `orgBrowserE2E.yml`, `lwcPlaywrightE2E.yml`, etc.) 2. **Filter target workflow** — add `--grep "Your Test Title"` to the test run command in the workflow you care about 3. **Optional** — skip org setup steps not needed for your test (e.g. minimal/non-tracking orgs) 4. **Restore** — remove branch from `branches-ignore`, remove `--grep`, uncomment skipped steps ## Test Controller Native Surfaces When testing native Test Controller surfaces (Test Explorer, Test Results panel): - **Test Results panel**: Wait for tab visibility, then assert Pass Rate text (e.g., `getByText(/Pass Rate/i)`) - **Tree items**: Assert aria-label contains expected decoration (e.g., `toHaveAttribute('aria-label', /Passed/i)` for completed tests) - **Locators**: Use `TEST_RESULTS_TAB = 'a.action-label[aria-label="Test Results"]'` to target panel tab reliably ## Reliable Assertions for Async Operations For desktop-only tests, prefer durable success signals over flaky UI assertions: - **Avoid**: `vscode.window.showInformationMessage` toasts auto-dismiss in seconds; `notification-list-item` assertions are racy - **Prefer**: Poll on-disk artifacts (e.g., generated files) with exponential backoff. Example: `waitForEsrFile` checks `fs.access` repeatedly until artifact appears or timeout. - Pattern: Create a helper that polls `fs.access` or `fs.stat` with `Date.now() < deadline` loop; throw on timeout with clear error message ## References - https://playwright.dev/docs - Playwright docs