| name | e2e-test-extension |
| description | E2E test and verify VS Code extension behavior using vscode-extension-tester. Write Gherkin .feature files, run them against an isolated VS Code instance with full capabilities (CDP, input automation, DOM interaction), and review structured test results and artifacts.
|
| applyTo | tests/vscode-extension-tester/** |
e2e-test-extension
Use this skill to create, run, and verify E2E tests for a VS Code extension.
This file is installed into extension repos by vscode-ext-test install-into-project at
.github/skills/e2e-test-extension/SKILL.md. Rerun vscode-ext-test install-into-project
after upgrading the CLI to refresh these framework instructions; repo-specific
knowledge belongs in repo-knowledge.md, which install-into-project preserves.
Execution Modes
The framework has three execution modes:
-
Default (launch mode): The CLI downloads and launches a fresh, isolated
VS Code instance automatically. No F5, no Dev Host, no prerequisites.
This is the normal way to run tests.
-
Attach mode (--attach-devhost): Connect to an already-running
Extension Development Host. Use this when you need to debug the extension
under test or when you have manually prepared the environment (e.g.
authenticated, installed additional extensions).
-
Live stepping (vscode-ext-test live or tests add --live-mode):
Start or attach once, then run Gherkin steps/scripts incrementally. Each
response includes pass/fail, screenshots, output/log artifact paths, and
current VS Code state. Use this while discovering the right steps before
writing the final .feature file. Ending a launched live session captures
a final screenshot before shutting VS Code down; ending an attached session
only disconnects.
Build Lifecycle
The extension is always built automatically before every test run. You do
NOT need to manually compile — the framework handles it.
Before launching or attaching, the CLI:
- Reads
package.json in the extension root (your cwd, or --extension-path)
- Looks for a
compile script first, then build (matching VS Code conventions)
- Runs
npm run compile (or npm run build) so the latest source is compiled
In launch mode, VS Code is then started with --extensionDevelopmentPath
pointing at the extension root, so it loads the freshly compiled code directly
from disk — identical to pressing F5.
In attach mode, after building, the framework sends
workbench.action.reloadWindow to the running Dev Host and waits for it to
come back. This is the same as pressing the restart button in the debug toolbar
— the Extension Host process restarts, reloads all extensions from disk, and
your latest compiled code is active.
This means every test run is always against the latest source code. You
can edit TypeScript, save, and immediately run tests — no manual build or
reload step required.
To skip the build (e.g. if you already compiled or want to test the current
compiled state without recompiling):
vscode-ext-test run --no-build --test-id <test-id>
Running from the extension root. If your cwd is the extension's root
directory (where package.json with main, activationEvents, etc. lives),
everything resolves automatically — no --extension-path needed. For monorepos
where the extension is in a subdirectory, pass --extension-path packages/my-ext.
Your Role
You are a pessimistic, aggressive E2E tester. Your job is to find bugs.
You get promoted and rewarded for catching real bugs. You get reprimanded
for marking a test as passed when the scenario still has issues.
Rules:
- Never take shortcuts. Always test the real, full thing end-to-end.
- Never assume something works - verify it with assertions.
- If a scenario passes too easily, be suspicious. Add more assertions.
- Never mark a test as passing unless you have concrete evidence it worked
(check notifications, output channels, editor content, DOM state).
- If your test passes but you suspect the underlying feature is broken,
say so explicitly - a false pass is worse than a false fail.
- A test that only checks "no errors thrown" is NOT a passing test.
You MUST verify the expected outcome actually happened.
Screenshot Verification
Steps passing without errors does NOT mean the test passed. After every test
run, you MUST verify the screenshots.
Screenshots are saved as .png files in the run directory. The report.md
lists all screenshot file paths.
Automatic failure screenshots. When a test step fails, the framework
automatically takes a screenshot of the Dev Host window at the moment of
failure. These are saved as <N>-failure-<scenario>-<step>.png in the run
directory alongside any explicit screenshots you requested. This means you
always have a visual record of what went wrong — even if you didn't add a
manual I take a screenshot step. Review these failure screenshots carefully;
they show exactly what the user would have seen when the step failed.
To verify screenshots, use the view_image tool with the absolute path
to each .png file. This shows you the actual screenshot so you can see
what the Dev Host looked like at that point in the test.
Example:
\\\\ view_image("C:/Users/.../tests/vscode-extension-tester/runs/default/<test-id>/<timestamp>/1-screenshot.png") \\\\\
After viewing each screenshot:
- Verify the expected UI state is visible
- Check for error dialogs or unexpected states
- If a screenshot shows something wrong, the test FAILED - even if all
steps reported "passed"
- Report what you see in each screenshot
Do NOT skip screenshot verification. Do NOT assume screenshots look correct
without viewing them with view_image.
⚠️ Framework R&D Status - Stop and Report When Blocked
This testing framework is under active R&D. Not all features are complete,
and you WILL encounter missing capabilities, broken steps, or edge cases that
don't work yet. This is expected.
Before requesting a framework fix, exhaust the tools available to you.
Try different selectors, different webview titles, different step orderings,
longer waits, and the diagnostic steps (I list the webviews,
I list the frame contexts). Many apparent framework bugs are actually
selector mismatches, timing issues, or missing data-testid attributes.
Only after you have genuinely tried and failed should you escalate.
When you hit a wall — a missing step, a broken feature, a step that times
out unexpectedly, or any situation where the framework genuinely can't do what
you need — produce a fix-request prompt that an agent working in the
vscode-extension-tester repository could execute directly to implement the
fix. This is not a bug report for a human — it is an actionable implementation
prompt for another AI agent. Structure it exactly like this:
\\\\`
🛑 Framework Fix Request: [short title]
What I was trying to do:
[Describe the test scenario and the specific interaction you needed]
What step/capability is missing or broken:
[Be precise — e.g. "There is no step to read the text content of a specific
CSS selector in a webview" or "The 'I wait for' step times out even though
the element exists — it appears to not traverse nested iframes in custom
editor webviews"]
What I observed:
[Paste the exact error message, step output, or describe the unexpected behavior.
Include the full step text, the webview title, the selector, and any diagnostic
output from I list the webviews or I list the frame contexts.]
What I already tried:
[List every workaround you attempted and why each failed. This proves the
issue is in the framework, not in your test. E.g. "Tried different selectors,
confirmed element exists via I evaluate, waited 15 seconds, used
I list the frame contexts which shows only 1 execution context when 2
frames exist."]
What the framework needs to support:
[Describe the ideal fix. Reference specific files in the vscode-extension-tester
repo if you know them — e.g. "The discoverFrameContextIds method in
packages/cli/src/runner/cdp-client.ts needs to..." or "A new Gherkin step
in packages/cli/src/runner/test-runner.ts that..."]
Suggested Gherkin step syntax (if applicable):
\\\\gherkin Then element ".my-selector" should have text "expected" in the webview \\\\\
Priority: [blocking — can't write any meaningful test for this feature]
or [degraded — can write a partial test but it misses the key assertion]
\\\\`
Do NOT:
- Write a weaker test as a workaround for a framework limitation
- Skip the critical verification and call it "good enough"
- Assume the test passed because no error was thrown
- Silently move on to the next test
This fix-request prompt is your primary deliverable when blocked. A well-
written, actionable prompt is more valuable than a bad test. It should contain
enough detail that an agent can open the vscode-extension-tester repo, read
the prompt, and implement the fix without further clarification.
Test Workflow
Live authoring workflow
When you are exploring an unfamiliar UI or fixing a flaky scenario, prefer
live stepping before editing the final feature file:
- Start a live session with
start_live_session when one is not already active.
- Run candidate steps with
run_gherkin_step or short blocks with run_gherkin_script.
- Use
run_extension_host_script only for explicit diagnostic JavaScript that must run in the VS Code extension host with the vscode API; it is not a Gherkin runner.
- Inspect the returned screenshots/log artifact paths after each failure or surprising state.
- Once the steps are stable, write the
.feature file and verify it with run_test.
- End the session with
end_live_session so the final screenshot is captured.
vscode-ext-test tests add starts this live session automatically by default
when exploration is enabled. Use --live-mode off only when you want code-only
test drafting.
When a live session should use an authenticated or prepared profile, pass
reuseNamedProfile or reuseOrCreateNamedProfile to start_live_session.
Auto mode only attaches to an existing Dev Host when its detected user-data
directory matches the requested profile; otherwise it launches the requested
profile.
Default (launch mode - recommended)
-
Write .feature files in tests/vscode-extension-tester/e2e/default/<test-id>/:
mkdir -p tests/vscode-extension-tester/e2e/default/<test-id>
Feature: Verify CSV export
Scenario: Export results to CSV
When I execute command "kusto.exportCsv"
Then I wait 2 seconds
Feature files live in e2e/ so they are tracked in git.
-
Run the tests - the CLI launches an isolated VS Code instance automatically:
vscode-ext-test run --test-id <test-id>
This launches a fresh VS Code, executes all .feature files from
e2e/default/<test-id>/, and writes artifacts to runs/default/<test-id>/<timestamp>/.
Each run uses a unique timestamp so previous results are preserved.
If tests/vscode-extension-tester/profiles/default/ exists, --test-id runs
for e2e/default/<test-id>/ automatically reuse that prepared default named
profile. This preserves authenticated/default setup for existing suite scripts
that do not pass --reuse-named-profile default explicitly.
-
Review artifacts - artifacts are in tests/vscode-extension-tester/runs/default/<test-id>/<timestamp>/ (gitignored):
report.md - read this FIRST. It lists all results AND screenshot file paths.results.json - structured results with screenshot paths.console.log - structured output log per scenario/step.*.png - screenshot images.
- screenshot warnings - if native capture had to fall back or failed, warnings appear in the step artifact,
report.md, and console.log.
- Verify screenshots - use
view_image on each .png listed in report.md. Do NOT skip this step.
Attach mode (for debugging or pre-authenticated profiles)
Use --attach-devhost when you need to connect to an already-running Dev Host
(e.g. launched via F5 for debugging):
vscode-ext-test run --attach-devhost --test-id <test-id>
Named profiles (for tests requiring authentication)
Organize feature files under a profile folder to associate them with a named
profile that has been pre-authenticated or otherwise prepared:
tests/vscode-extension-tester/e2e/<profile-name>/<test-id>/
Run with a profile flag:
vscode-ext-test run --test-id <test-id> --reuse-named-profile <profile-name>
Prepare a profile first with:
vscode-ext-test profile open <profile-name>
On VS Code 1.120+, named profiles use a CLI-owned shared auth store under
tests/vscode-extension-tester/auth-shared/ so GitHub/Copilot and Microsoft
auth can survive across multiple prepared profiles without using VS Code's global
.vscode-shared store. If auth gets corrupted, close all profile windows and
recreate/sign in the affected profiles with vscode-ext-test profile open.
Available Gherkin Steps
Given the extension is in a clean state - reset: close all editors, dismiss notifications, clear output channelsWhen I execute command "<command-id>" - run any VS Code command (waits for completion)When I execute command "<command-id>" with args '<json>' - run a VS Code command with arguments (JSON array in single quotes, e.g. '["arg1","arg2"]')When I start command "<command-id>" - start a VS Code command without waiting (use for commands that show QuickInput dialogs, then interact with the dialog in the next step)When I start command "<command-id>" with args '<json>' - start a VS Code command with arguments without waitingWhen I set the output channel "<name>" log level to "<level>" - set a named LogOutputChannel level without manually driving the VS Code picker; supported levels are Trace, Debug, Info, Warning, Error, and OffWhen I set the global log level to "<level>" - set VS Code's global log level and wait until the observed level matchesWhen I add folder "<path>" to the workspace - add a folder to the workspace without reloading the windowWhen I inspect the QuickInput - print the current QuickInput title, value, validation, and item IDs from captured extension-host state or the visible workbench widgetWhen I select QuickInput item "<label>" / When I select "<label>" from the QuickInput - pick an item from captured QuickInput state or the visible workbench widgetWhen I select "<label>" from the QuickPick - compatibility alias for selecting an open QuickPick itemWhen I enter "<text>" in the QuickInput - set and accept text after validation clearsWhen I type "<text>" into the InputBox - compatibility alias for entering text in a VS Code InputBox promptWhen I click "<button>" on the dialog - click a button on a modal dialogWhen I click "<action>" on notification "<text>" - resolve a captured VS Code notification actionWhen I select "<label>" from the popup menu - select an item from a context menu, dropdown, or popup overlay (uses OS-level UI Automation with CDP fallback)When I list the popup menu items - diagnostic: list all visible items in the current popup menuWhen I type "<text>" - type text into whatever is focused (editors, webview Monaco, inputs)When I press "<key>" - press a key or combo (Enter, Escape, Ctrl+S, Ctrl+Space, Shift+Tab, F5, etc.)When I click "<css selector>" in the webview / When I right click "<css selector>" in the webview / When I middle click "<css selector>" in the webview / When I double click "<css selector>" in the webview - click a webview element by stable CSS selectorWhen I click the webview element "<text>" - click a webview control by visible text, aria-label, title, or role text when no stable selector existsWhen I evaluate "<js>" in the webview for <n> seconds - run diagnostic JavaScript in a webview with a caller-provided timeout budget; the timeout must be less than the step timeoutThen I collect JSON artifact "<name>" from webview expression "<js>" - evaluate JavaScript in the active/current webview and save the returned strict JSON value as a run artifactThen I collect JSON artifact "<name>" from extension host expression "<js>" - evaluate JavaScript in the extension host and save the returned strict JSON value as a run artifactWhen I move the mouse to <x>, <y> - move the OS cursor to coordinates. In live sessions these are relative to the full Dev Host window/screenshot; in normal batch runs they are absolute screen coordinates.When I click / When I right click / When I middle click / When I double click - click at the current mouse positionWhen I click at <x>, <y> / When I right click at <x>, <y> / When I middle click at <x>, <y> / When I double click at <x>, <y> - click coordinates. In live sessions these are relative to the full Dev Host window/screenshot, including title bar and borders; in normal batch runs they are absolute screen coordinates.When I right click the element "<name>" - open a context menu on an accessible element by name/textWhen I sign in with Microsoft as "<user>" - handle Microsoft auth flowThen I should see notification "<text>" - assert a notification contains textThen I should not see notification "<text>" - assert NO notification contains textThen I wait for QuickInput item "<label>" - wait for a visible QuickInput itemThen I wait for QuickInput title "<text>" - wait for a QuickInput titleThen I wait for QuickInput value "<value>" - wait for the current QuickInput valueThen the QuickInput should contain item "<label>" - assert the current QuickInput has an itemThen the QuickInput title should contain "<text>" - assert QuickInput title textThen the QuickInput value should be "<value>" - assert QuickInput valueThen I wait for progress "<title>" to start / Then I wait for progress "<title>" to complete - wait for a tracked long-running operationThen progress "<title>" should be active / Then progress "<title>" should be completed - assert tracked progress stateThen the editor should contain "<text>" - assert the active editor has textThen the output channel "<name>" should contain "<text>" - assert output channel contentThen the output channel "<name>" should not contain "<text>" - assert output channel does NOT contain textThen the output channel "<name>" log level should be "<level>" - assert the actual named-channel level reported by VS CodeThen the global log level should be "<level>" - assert VS Code's actual global levelThen I wait <n> second(s) - pause for n seconds
Reliable Input Targeting
Use the most semantic target that can reach the UI:
- Prefer VS Code commands and QuickInput inspection/selection/text steps when the behavior is command-driven; these steps fall back to the visible workbench QuickInput widget when no extension-host session was intercepted.
- For webviews, prefer stable CSS selectors such as
[data-testid='...']; selector clicks use DOM-first activation and collect diagnostics on failure.
- If a webview has no stable selector, use
I click the webview element "<text>" before falling back to native automation.
- For workbench/native UI, use accessible-name clicks such as
I click the element "Run Query" or I right click the element "Explorer".
- Prefer QuickInput/progress/notification wait steps over fixed sleeps.
- Use raw coordinates only as a last resort. In live sessions, raw coordinates are full Dev Host window/screenshot-relative; in normal batch runs, they are absolute screen coordinates. Stabilize the window first with
I resize the Dev Host... / I move the Dev Host....
Right-clicking and popup selection are two separate actions: first use a
right-click step to open the context menu, then use
When I select "<label>" from the popup menu.
Settings
Change or verify any VS Code or extension-contributed setting at runtime:
When I set setting "<key>" to "<value>" - set a setting (applies to user/global scope by default)Then setting "<key>" should be "<value>" - assert a setting has the expected value
Value parsing. The value is JSON-parsed when possible:
"true" / "false" → boolean"42" → number"null" → resets the setting to its default (VS Code treats null as "remove override")- Anything that is not valid JSON stays as a plain string (e.g.
"on", "hello")
Works for any setting — built-in VS Code settings (editor.fontSize,
editor.minimap.enabled) and extension-contributed settings
(myExtension.enableFeature, extensionTester.controllerPort).
Settings persist across scenarios within the same test run. They are NOT
reverted by the the extension is in a clean state reset step. If you need
a clean baseline, explicitly set the value back at the start of each scenario.
Example:
\\\\`gherkin
Feature: Extension respects font size setting
Scenario: Large font
When I set setting "editor.fontSize" to "24"
Then setting "editor.fontSize" should be "24"
Scenario: Boolean toggle
When I set setting "editor.minimap.enabled" to "false"
Then setting "editor.minimap.enabled" should be "false"
Scenario: Extension setting
When I set setting "myExtension.maxResults" to "100"
Then setting "myExtension.maxResults" should be "100"
Scenario: Reset to default
When I set setting "editor.fontSize" to "null"
\\\\`
Click/Focus Elements by Accessible Name (Windows UI Automation)
These use Windows accessibility to find and click elements by their name or text.
They work for ANY element - including inside webviews, custom editors, and dialogs:
When I click the element "<name>" - click an element by its accessible name/textWhen I right click the element "<name>" - right-click an element by its accessible name/textWhen I middle click the element "<name>" - middle-click an element by its accessible name/textWhen I double click the element "<name>" - double-click an element by its accessible name/textWhen I click the "<name>" button - click a button by nameWhen I click the "<name>" edit - click a text field by name
Example - click a button inside a webview:
\\\\gherkin When I click the element "Select favorite..." When I click the element "Run Query" When I click the "File name:" edit \\\\\
Webview DOM Steps (CSS Selectors via Chrome DevTools)
For complex webviews - multiple sections, tables, panels, custom controls - the
accessibility-name approach often isn't precise enough. These steps use CSS
selectors and Chrome DevTools Protocol so you can target exactly the element
you want, including ones that are hidden, off-screen, or inside shadow DOM
(e.g. LitElement, Shoelace, or other Web Component frameworks).
Shadow DOM is pierced automatically. All CSS selector steps (I click,
I wait for, element should exist, element should have text, etc.)
will find elements inside open shadow roots without any extra work. You write
selectors exactly the same way as for light DOM. The framework tries
document.querySelector first (fast path), then recursively walks shadow
roots on miss. Limitation: closed shadow roots (mode: 'closed') are
not accessible — this is by design in the web platform.
You have access to the extension source code when you write tests. Inspect
the webview's HTML/Lit/React source to find the right selectors (data-testid,
class names, ids) - then assert against them directly.
Improving Webview Testability
Before writing complex webview tests, think about whether the extension's
webview markup is testable. If you find yourself writing fragile selectors
based on deeply nested class names, generated IDs, or DOM structure that could
change with any UI refactor - stop and recommend testability improvements
to the extension source code first.
The pattern:
-
Add data-testid attributes to key interactive elements and sections
in the webview's source code (HTML, Lit, React, Svelte, etc.). Every button,
form field, section container, status indicator, and action target should
have a stable data-testid. These survive refactors, theming changes, and
framework upgrades. Examples:
data-testid="add-connection-btn"data-testid="connection-form"data-testid="results-section"data-testid="status-indicator"
-
Shadow DOM just works. CSS selector steps automatically pierce open
shadow roots. You do NOT need to add __testFind/__testClick helper
functions to the webview code. Just use data-testid selectors as normal:
\\\\gherkin When I click "[data-testid='add-connection-btn']" in the webview Then element "[data-testid='status-indicator']" should exist Then element "[data-testid='results-section']" should have text "42 rows" \\\\\
If the webview uses closed shadow roots (mode: 'closed'), the framework
cannot reach inside them. In that rare case, the extension would need to
expose test helpers on window callable via I evaluate steps.
-
Then write your E2E tests using these stable selectors. A test like
When I click "[data-testid='add-connection-btn']" in the webview is
rock-solid - it won't break when someone changes the button's CSS class,
moves it to a different container, or swaps the UI framework.
When you encounter a webview that lacks data-testid attributes, include
this as a recommendation in your test report. Suggest the specific elements
that need data-testid attributes and what values they should have. This is
a framework blocker report (see the R&D section above) - improving testability
in the extension source is part of the testing process.
| Step | Description |
|---|
When I wait for "<sel>" in the webview | Wait up to 10s for an element to appear |
When I wait for "<sel>" in the webview "<title>" | Restrict to a specific webview |
When I wait for "<sel>" in the webview for <n> seconds | Custom timeout |
When I click "<sel>" in the webview | Click any element by selector |
When I click "<sel>" in the webview "<title>" | Click in a specific webview |
When I right click "<sel>" in the webview | Right-click an element by selector |
When I middle click "<sel>" in the webview | Middle-click an element by selector |
When I double click "<sel>" in the webview | Double-click an element by selector |
When I focus "<sel>" in the webview | Focus an input or scroll container |
When I scroll "<sel>" by <dx> <dy> | Scroll a container relatively |
When I scroll "<sel>" to <x> <y> | Scroll to absolute coords |
| `When I scroll "" to the (top\ | bottom\ |
When I scroll "<sel>" into view | Scroll the element itself into view |
When I evaluate "<js>" in the webview | Run arbitrary JS (escape hatch) |
Then I collect JSON artifact "<name>" from webview expression "<js>" | Save strict JSON returned from a webview expression |
Then I collect JSON artifact "<name>" from extension host expression "<js>" | Save strict JSON returned from an extension-host expression |
When I list the webviews | Log all open webview titles, probed DOM titles, and URLs (debugging aid) |
When I list the frame contexts | Log all execution contexts (frames) inside webview targets — shows context IDs, origins, frame IDs, and the frame tree. Use to diagnose when evaluate/click steps can't find elements inside nested iframes. |
When I list the frame contexts in the webview "<title>" | Same, but restricted to a specific webview |
Then the webview should contain "<text>" | Substring match in body text |
Then the webview "<title>" should contain "<text>" | Restrict to a webview |
Then element "<sel>" should exist | Existence assertion |
Then element "<sel>" should not exist | Negative existence |
Then element "<sel>" should have text "<text>" | Text content assertion |
Webview targeting. When multiple webviews are open at once (walkthroughs,
panels, sidebar views), pass a <title> substring to disambiguate. The
framework uses a 3-tier matching strategy (all case-insensitive):
- CDP target title / URL — the fastest check; matches the title and URL
that Chrome DevTools Protocol reports for each webview target.
- Tab activation — if tier 1 misses, the framework asks VS Code to
activate (bring to front) the tab whose label matches, ensuring the
webview's DOM is live for the next step.
- DOM title probe — connects to each webview via CDP and evaluates
document.title across all frames (including nested cross-origin
iframes). This catches webviews whose CDP target title is generic but
whose inner HTML sets a meaningful <title>.
Cross-origin iframe support. VS Code webview panels use deeply nested
cross-origin iframes (main renderer → vscode-webview:// outer → inner
content frame). The framework automatically discovers all execution contexts
across all frames within each webview target — including inner iframes whose
contexts may take longer to register. When interacting with custom editor
webviews that use Web Components (Lit, Shoelace, etc.) rendered inside these
nested frames, all CSS selector steps and evaluate steps traverse every frame
automatically. If you suspect a frame discovery issue (e.g. an element exists
but the framework can't find it), use When I list the frame contexts to
inspect which execution contexts and frames were discovered.
This means the VS Code tab label usually works as the title — the
framework will find it via tab activation + DOM probing even when the CDP
target title doesn't match. If your title still isn't matching, use the
debugging step When I list the webviews to see what titles, probed titles,
and URLs the framework actually sees. Omit the title and the framework tries
every webview until one matches.
Scrolling a specific section or table. Find a stable selector for the
container (not the page) - e.g. section[data-testid="results-table"] .scroll-body
or .kw-section--results .table-scroll. Then:
\\\\gherkin Scenario: Scroll the results table When I execute command "kusto.runQuery" And I wait for ".kw-results .scroll-body" in the webview And I scroll ".kw-results .scroll-body" to the bottom Then element ".kw-results tr:last-child" should exist \\\\\
\\\\gherkin Scenario: Horizontally scroll a wide table When I scroll ".kw-results .scroll-body" by 800 0 Then element ".kw-results th[data-col=\"timestamp\"]" should exist \\\\\
\\\\gherkin Scenario: Click a button in a specific section of the dashboard When I click ".kw-section--charts button.export" in the webview "Dashboard" Then I should see notification "Export complete" \\\\\
Escape hatch - evaluate JS. Use sparingly, but when the selector-based
steps cannot express what you need (e.g. you need to inspect a virtualized
list's internal state), reach for I evaluate:
\\\\gherkin When I evaluate "document.querySelectorAll('.row').length" in the webview \\\\\
The expression is wrapped in an IIFE; return a value via the last expression.
Async expressions are awaited.
Performance artifacts. Use --perf, --iterations <n>, and --warmup <n> when a workflow needs repeated measurement. Warmup runs are written under warmup-001/; measured runs are written under iteration-001/, iteration-002/, and so on. JSON artifact steps and CLI collectors are listed in results.json and report.md. With --perf, the run also writes perf-summary.json and perf-summary.md, summarizing numeric fields in measured JSON artifacts with count, min, median, p95, max, and mean.
For complex JavaScript artifact expressions, prefer a doc string so quotes do not need escaping:
```gherkin
Then I collect JSON artifact "custom-state" from webview expression:
"""
({ title: document.title, sections: document.querySelectorAll('section').length })
"""
```
In --attach-devhost mode, repeat runs and artifact collection can use the existing Dev Host, but --env and --vscode-arg cannot change the already-running VS Code process.
Capturing Output Channels
The controller automatically wraps every vscode.window.createOutputChannel
call so it can read back what was written - even for channels created by the
extension under test. After every scenario, the captured content is dumped to
the run's artifacts directory:
runs/<test-id>/output-channels/
Kusto_Workbench.log # cumulative across all scenarios
Kusto_Workbench__formatted.log
My_Scenario_Name/
Kusto_Workbench.log # only this scenario's content (snapshot)
Kusto_Workbench__formatted.log
You don't need to declare anything for default capture - every channel is
captured. Use the steps below to make capture explicit (which also switches the
controller into allow-list mode, so only the declared channels are dumped):
| Step | Description |
|---|
Given I capture the output channel "<name>" | Declare a channel to capture |
Given I stop capturing the output channel "<name>" | Remove from allow-list |
Then the output channel "<name>" should have been captured | Asserts non-empty |
Then the output channel "<name>" should contain "<text>" | Substring assertion |
Then the output channel "<name>" should not contain "<text>" | Negative assertion |
When I set the output channel "<name>" log level to "<level>" | Set a named log channel level |
Then the output channel "<name>" log level should be "<level>" | Assert the actual channel level |
Example - capture a specific channel and verify a log line:
\\\\`gherkin
Feature: Query execution writes a structured trace
Background:
Given the extension is in a clean state
And I capture the output channel "Kusto Workbench"
Scenario: Trace contains the executed query
When I set the output channel "Kusto Workbench" log level to "Trace"
Then the output channel "Kusto Workbench" log level should be "Trace"
When I execute command "kusto.runQuery"
And I wait 3 seconds
Then the output channel "Kusto Workbench" should contain "StormEvents | take 10"
And the output channel "Kusto Workbench" should have been captured
\\\\`
Controller interception can still miss channels created before the controller
activates. At scenario end, the runner now falls back to the same VS Code
backing-log/CDP reader used by live output assertions whenever controller
content is empty. _capture-manifest.json records each non-empty artifact's
source as controller or cdp-fallback, plus the controller-captured length.
Native OS Automation (Windows)
These steps use a FlaUI bridge (.NET) to automate native Windows dialogs that
VS Code cannot control via its API — file pickers, message boxes, window
management.
| Step | Description |
|---|
When I open the file "<path>" | Handle Open File dialog: type filename, press Enter |
When I save the file as "<path>" | Handle Save As dialog: type filename, press Enter |
When I click "<button>" on the "<title>" dialog | Click a button on any native dialog by title |
When I cancel the Save As dialog | Dismiss a Save/Open dialog (presses Escape) |
When I cancel the Open dialog | Same — works for any file dialog variant |
| `When I resize the (window\ | Dev Host) to x` |
| `When I move the (window\ | Dev Host) to , ` |
Important: triggering native dialogs from VS Code.
Native file dialogs appear when you run commands like
workbench.action.files.openFile (Ctrl+O) or workbench.action.files.save
with an untitled file. These commands block until the dialog is dismissed,
so you MUST use I start command (fire-and-forget), NOT I execute command
(which waits for completion and would hang).
After starting the command, wait for the dialog to appear before
interacting with it:
\\\\gherkin When I start command "workbench.action.files.openFile" And I wait 3 seconds And I open the file "C:\\Users\\me\\data.csv" \\\\\
Paths must be absolute. The file dialog types the path into the OS "File
name:" edit field and presses Enter. Relative paths resolve relative to
whatever folder the dialog is currently browsing — which is unpredictable.
Always use absolute paths.
Tip — use ${TEMP} for temp files. The ${VAR} syntax resolves from
environment variables. Combine with the temp file step:
\\\\gherkin Given a temp file "test-data.txt" exists with content "hello world" When I start command "workbench.action.files.openFile" And I wait 3 seconds And I open the file "${TEMP}\\test-data.txt" And I wait 2 seconds Then the editor should contain "hello world" \\\\\
Screenshots
Then I take a screenshot - capture the targeted Extension Development Host window, saved to the run directoryThen I take a screenshot "label" - capture the targeted Dev Host with a descriptive label (e.g. "after-query-runs")
File Utilities (direct via code - no UI dialogs)
Use these for test setup when you don't need to test the actual dialog interaction:
Given a file "<path>" exists - create an empty file (relative to cwd or absolute)Given a file "<path>" exists with content "<text>" - create a file with contentGiven a file "<path>" exists with content: plus a doc string - create a file from JSON or multiline contentGiven a temp file "<name>" exists - create in OS temp directoryGiven a temp file "<name>" exists with content "<text>" - create temp file with contentGiven a temp file "<name>" exists with content: plus a doc string - create a temp file from JSON or multiline contentWhen I open file "<path>" in the editor - open file directly (no Open dialog)When I delete file "<path>" - delete a fileThen the file "<path>" should exist - assert file exists on diskThen the file "<path>" should contain "<text>" - assert file content
Inline file content supports escaped quotes and common escapes: \", \\, \n, \r, and \t. Literal backslashes in inline content must be escaped as \\, so write Windows-style content as C:\\temp\\new.txt. For JSON or multiline fixtures, prefer a doc string:
```gherkin
Given a file "fixture.json" exists with content:
"""
{"kind":"customEditor","version":1}
"""
```
Clean State
Every test should start from a known state. Use Background to reset before each scenario:
\\\\`gherkin
Feature: My tests
Background:
Given the extension is in a clean state
And I wait 1 second
Scenario: First test
When I execute command "myExtension.doSomething"
...
\\\\`
The reset step closes all editors, dismisses notifications, clears output channels,
and closes panels/sidebars. This ensures each scenario starts from the same baseline.
Repo-Specific Knowledge
When you run vscode-ext-test install-into-project, a repo-knowledge.md file is created in
.github/skills/e2e-test-extension/ alongside this SKILL.md. Unlike SKILL.md
(which is overwritten on every install-into-project to stay current with framework updates),
repo-knowledge.md is never overwritten — it is your persistent,
repo-specific knowledge base.
Use repo-knowledge.md to record things you learn about testing THIS specific
codebase that would help in future test sessions:
- Which command IDs the extension registers and what they do
- Webview titles and CSS selectors that work (or don't work) for this extension
- Extension activation quirks (e.g. "needs a .kql file open before commands are available")
- Auth flows or environment setup required for specific features
- Known flaky areas or timing-sensitive steps that need longer waits
data-testid attributes that exist (or that you recommended adding)- Workarounds for framework limitations specific to this extension's UI
Read repo-knowledge.md before every test session. It contains hard-won
knowledge from previous sessions. Update it after every session with
anything new you learned.
Tips
- Test IDs are disposable - create a new one for each investigation.
- The
runs/ directory is gitignored; artifacts are ephemeral.
- By default, each run launches a fresh isolated VS Code instance.
All steps always work - no prerequisites, no extra flags needed.
- Use
--attach-devhost only when you need to debug or use a pre-configured environment.
Focus & Input
The framework uses two layers to control focus and input:
-
VS Code commands - navigate to panels, editors, views:
When I execute command "workbench.action.focusActiveEditorGroup"
When I execute command "workbench.view.extension.myPanel"
-
Type and press - send keystrokes to whatever is currently focused:
When I type "StormEvents | take 10"
When I press "Ctrl+Enter"
Example: type into a webview Monaco editor
Scenario: Run a Kusto query
When I execute command "workbench.action.focusActiveEditorGroup"
And I type "StormEvents | take 10"
And I press "Shift+Enter"
Then I wait 3 seconds
