| name | openspec-e2e |
| description | Run Playwright E2E verification for an OpenSpec change. Use when the user wants to validate that the implementation works end-to-end by running Playwright tests generated from the specs. |
| license | MIT |
| compatibility | Requires openspec CLI, Playwright (with browsers installed), /browse (gstack, for exploration), and @playwright/mcp (globally installed via `claude mcp add playwright npx @playwright/mcp@latest`, for test execution + Healer). |
| metadata | {"author":"openspec-playwright","version":"2.25"} |
Input
- Change name:
/opsx:e2e <name> or /opsx:e2e all (full app exploration, no OpenSpec needed)
- Specs:
openspec/changes/<name>/specs/*.md (if change mode)
- Credentials:
E2E_USERNAME + E2E_PASSWORD env vars
Output
- Test file:
tests/playwright/changes/<name>/<name>.spec.ts
- Page Objects (all mode):
tests/playwright/pages/<Route>Page.ts
- Auth setup:
tests/playwright/auth.setup.ts (if auth required)
- Report:
openspec/reports/playwright-e2e-<name>-<timestamp>.md
- App Bug Registry:
openspec/reports/app-bug-registry.md (cumulative, per-project)
- Test plan:
openspec/changes/<name>/specs/playwright/test-plan.md (change mode only)
Architecture
Two modes, same pipeline:
| Mode | Command | Route source | Output |
|---|
| Change | /opsx:e2e <name> | OpenSpec specs | changes/<name>/<name>.spec.ts |
| All | /opsx:e2e all | sitemap + homepage crawl | pages/*.ts (Page Objects) |
Both modes update app-knowledge.md and app-exploration.md. Spec files are independent per change โ openspec-pw run <name> runs only changes/<name>/<name>.spec.ts.
โ ๏ธ Full regression is opt-in only. Default: openspec-pw run <name> โ one spec file. Do NOT run npx playwright test (no file), --only-changed, or any command that executes multiple .spec.ts files unless the user explicitly requests it. This includes running the same command twice across different changes to simulate regression.
Role mapping: Planner (Step 4โ5) โ test-plan.md; Generator (Step 6) โ .spec.ts + Page Objects; Healer (Step 9) โ repairs failures via MCP.
Setup / Teardown
Playwright supports two approaches for global lifecycle hooks. openspec-playwright uses project dependencies (the recommended approach) for full feature support.
Comparison
| Feature | Project Dependencies | globalSetup/globalTeardown |
|---|
| HTML report visibility | โ
Shown as project | โ Not shown |
| Trace recording | โ
Full support | โ Not supported |
| Playwright fixtures | โ
Fully supported | โ Not supported |
| Browser via fixture | โ
Automatic | โ Manual launch |
Current Implementation
Setup project (enabled by default):
tests/playwright/auth.setup.ts โ authenticates once, saves session to ./playwright/.auth/user.json- All test projects depend on setup via
dependencies: ['setup']
Teardown project (optional, disabled by default):
tests/playwright/global.teardown.ts โ runs AFTER all tests complete- Use for: database cleanup, uploaded file removal, cache invalidation
- Enable by uncommenting in
playwright.config.ts
When to Enable Teardown
Enable teardown when your tests create persistent data that should be cleaned up:
| Scenario | Action |
|---|
| Tests create database records | โ
Enable teardown, add DB cleanup |
| Tests upload files | โ
Enable teardown, add file cleanup |
| Tests only read data | โ No teardown needed |
| Tests use ephemeral/isolated environments | โ No teardown needed |
Enabling Teardown
- Copy template:
cp templates/global.teardown.ts tests/playwright/global.teardown.ts
- Customize cleanup logic in the file
- Uncomment in
playwright.config.ts:
projects: [
{ name: 'setup', testMatch: /.*\.setup\.ts/ },
{ name: 'teardown', testMatch: /global\.teardown\.ts/ },
{
name: 'chromium',
teardown: 'teardown',
},
],
Testing principles
UI first โ Test every user flow through the browser UI. E2E validates that users can accomplish tasks in the real interface, not just that the backend responds correctly.
็จๆทๆไฝ โ ๆต่งๅจ UI โ ๅ็ซฏ โ ๆฐๆฎๅบ โ UI ๅ้ฆ
API only as fallback โ See Mock data rule below. page.request is acceptable for pre-condition setup (preparing test data) and API-level mocking via page.route(). Every final assertion about visible UI state must use UI selectors โ never use page.request to assert something the user can see on screen.
Decision rule (per assertion):
Can the user SEE this on screen?
โ Yes โ MUST use: page.getByRole/ByLabel/ByText + expect()
โ No โ Record reason โ page.request acceptable
Business logic assertion rule โ numerical/calculated values MUST use API assertion:
UI assertions verify rendering correctness, not calculation correctness. A UI that correctly displays a wrong value will pass UI-only tests.
Is the assertion about a computed/counted/calculated value?
(e.g., balance, total, discount, count, percentage, score, ranking)
โ Yes โ Use page.request to fetch backend data โ assert the raw value
โ No โ UI assertion is sufficient
Mock data rule:
**Examples where API assertion is required:
await page.getByText('ยฅ800').click();
await expect(page.getByText('ๆป้้ข: ยฅ800')).toBeVisible();
const order = await page.request.get(`${BASE_URL}/api/orders/${orderId}`);
const body = await order.json();
expect(body.total).toBe(800);
await page.getByRole('button', { name: '็น่ต' }).click();
await expect(page.getByText('1 ไธช่ต')).toBeVisible();
const post = await page.request.get(`${BASE_URL}/api/posts/${postId}`);
const body = await post.json();
expect(body.likeCount).toBeGreaterThan(0);
await page.getByRole('button', { name: '็น่ต' }).click();
await expect(page.getByText('1 ไธช่ต')).toBeVisible();
await page.waitForResponse(r => r.url().includes('/api/like'));
const post = await page.request.get(`${BASE_URL}/api/posts/${postId}`);
expect((await post.json()).likeCount).toBeGreaterThan(0);
Never use API calls to replace routine UI flows. If a test completes in < 200ms, it is almost certainly using page.request instead of real UI interactions.
Steps
1. Select the change or mode
Change mode (/opsx:e2e <name>):
- Use provided name, or infer from context, or auto-select if only one exists
- If ambiguous โ
openspec list --json + AskUserQuestion
- Verify specs exist:
openspec status --change "<name>" --json
- If specs empty โ STOP: E2E requires specs. Use "all" mode instead.
"all" mode (/opsx:e2e all โ no OpenSpec needed):
- Announce: "Mode: full app exploration + Page Object discovery"
- Goal: Discover new routes, extract selectors, and build
pages/*.ts Page Objects โ accumulated asset for future Change tests
- Route discovery (in order):
- sitemap.xml:
$B goto ${BASE_URL}/sitemap.xml โ parse URLs
- Link extraction: Navigate to
${BASE_URL}/ โ $B js extracts all <a href>:
() => {
const origin = window.location.origin;
const links = Array.from(document.querySelectorAll('a[href]'));
return links
.map(a => a.href)
.filter(h => h.startsWith(origin) && !h.includes('/logout') && !h.includes('/api/'))
.map(h => new URL(h).pathname);
}
- Fallback common paths:
/, /login, /dashboard, /admin, /profile, /settings
Decision table โ route discovery fallback:
| Situation | Action |
|---|
sitemap.xml returns 200 with URLs | Parse all URLs โ extract pathname |
sitemap.xml returns 404/5xx | Skip โ use link extraction |
| Link extraction finds 0 links | Fall back to common paths |
| Common path returns 200 | Add to routes |
| Duplicate routes from multiple sources | Deduplicate by pathname |
- Persist routes: Write discovered routes to
app-knowledge.md โ Routes table. Replace the entire table (including header) with fresh data โ do not append.
- Group routes: Guest vs Protected (by attempting direct access)
2. Detect auth
Change mode: Read specs and extract functional requirements. Detect auth from keywords.
"all" mode: Detect auth by attempting to access known protected paths (e.g. /dashboard, /profile). If redirected to /login โ auth required.
Auth detection โ both modes (BOTH conditions required):
Condition A โ Explicit markers: "login", "signin", "logout", "authenticate", "protected", "authenticated", "session", "unauthorized", "jwt", "token", "refresh", "middleware"
Condition B โ Context indicators: Protected routes ("/dashboard", "/profile", "/admin"), role mentions ("admin", "user"), redirect flows
Exclude false positives: HTTP header examples (Authorization: Bearer ...) and code snippets do not count.
Confidence โ decision table:
| Confidence | Condition | Action |
|---|
| High | Multiple markers AND context indicators | Auto-proceed |
| Medium | Single marker, context unclear | Proceed + note in output |
| Low | No markers found | Skip auth, test as guest |
3. Validate environment
Run the seed test before generating tests:
npx playwright test tests/playwright/seed.spec.ts --project=chromium
Seed test initializes the page context โ it runs all fixtures, hooks, and globalSetup. Not just a smoke check: it also validates that auth setup, BASE_URL, and Playwright are fully functional.
If seed test fails: Stop and report. Fix the environment before proceeding.
4. Explore application
Explore to collect real DOM data before writing test plan. This eliminates blind selector guessing.
Prerequisites: seed test pass. BASE_URL must be verified reachable (see 4.1). If auth is required and auth.setup.ts already exists โ auth is ready. If auth is not yet configured โ use the workaround below (Option B for protected routes).
4.1. Verify BASE_URL + Read app-knowledge.md
- Verify BASE_URL:
$B goto <BASE_URL> โ if HTTP 5xx โ STOP: backend error. Fix app first.
- Read app-knowledge.md: known risks, project conventions
- Routes (from Step 1): use already-discovered routes โ no need to re-extract
4.2. Explore each route via /browse
For each route:
$B goto <url> โ $B console โ $B snapshot โ $B screenshot
Parallel Exploration (via openspec-pw explore)
For apps with many routes (>=5), use the dedicated CLI command for true parallel exploration:
openspec-pw explore --parallel 4
openspec-pw explore --dry-run
Why: $B uses a single shared Chromium instance with one active page โ Promise.allSettled on $B commands serializes execution and causes navigation state conflicts. openspec-pw explore launches N independent browser processes, each with its own Chromium context, for genuine parallelism.
After parallel exploration completes, read the updated app-exploration.md to continue with Step 4.3.
After navigating, check for app-level errors:
| Signal | Meaning | Action |
|---|
| HTTP 5xx or unreachable | Backend/server error | STOP โ tell user: "App has a backend error (HTTP ). Fix it, then re-run /opsx:e2e <name> to re-explore." |
| JS error in console | App runtime error | STOP โ tell user: "Page has JS errors. Fix them, then re-run /opsx:e2e <name> to re-explore." |
| HTTP 404 | Route not in app (metadata issue) | Continue โ mark โ ๏ธ route not found in app-exploration.md |
| Auth required, no credentials | Missing auth setup | Continue โ skip protected routes, explore login page |
| Suspicious network request | API returned 4xx/5xx | Continue โ mark โ ๏ธ API error: <endpoint> returned <code> in app-exploration.md |
Redirect / Refresh loop detection โ run after initial navigate:
// 1. Initial capture
$B goto <url>
$B wait --networkidle // wait for SPA hydration
$B js "window.location.href" // url1
$B wait 2000
// 2. Observe stability
$B js "window.location.href" // url2
$B console --errors
// 3. Detect
if (url1 !== url2) {
โ โ URL changed โ redirect loop
โ Is this route protected without valid auth?
โ Yes โ auth.setup.ts is broken โ fix auth first
โ No โ App middleware bug โ mark route โ skip, record as App Bug
}
if (console errors/warnings > 10) {
โ โ Excessive console errors โ page refresh / JS crash loop
โ Mark route โ skip, record as App Bug
}
Network monitoring: After navigating, use $B network to check for failed API calls. Failed requests (status โฅ 400) on a route indicate an API/backend issue โ record in app-exploration.md for reference.
For guest routes (no auth):
$B goto <url>
For protected routes (auth required):
// Option A: use existing storageState (recommended)
// Option B: navigate to /login first, fill form, then navigate to target
// Option C: $B cookie to set auth cookies directly
If credentials are not yet available:
- Skip protected routes โ mark
โ ๏ธ auth needed โ explore after auth.setup.ts
- Explore the login page itself (guest route) โ extract form selectors
- After auth.setup.ts runs, re-run exploration for protected routes
Wait for page stability:
- React 19 / Next.js App Router: use
page.waitForLoadState('networkidle') โ React 19 concurrent mode batches events asynchronously; 200-500ms timeouts are unreliable under resource contention
- Vue 2/3 / Angular / React 18 / Plain JS / jQuery:
waitForSelector(targetElement) is sufficient and faster โ DOM updates are synchronous; Playwright's actionability checks auto-wait correctly
- Prefer specific element waits (
waitForSelector) over generic load states
- Ready signal: heading, spinner disappears, or URL change
4.3. Parse the snapshot
From $B snapshot output, extract interactive elements for each route:
| Element type | What to capture | Selector priority |
|---|
| Buttons | text, selector | [data-testid] > getByRole > getByLabel > getByText |
| Form fields | name, type, label, selector | [data-testid] > name > label |
| Navigation links | text, href, selector | text > href |
| Headings | text content, selector | for assertions |
| Error messages | text patterns, selector | for error path testing |
| Dynamic content | structure โ row counts, card layouts | for data-driven tests |
| Special elements | type, selector, dimensions | for canvas/iframe/Shadow DOM test strategies |
4.3.1. Detect special elements
From $B snapshot + $B js, identify these special elements per route:
Special element detection matrix:
| Element | Snapshot signal | Evaluate supplement | Exploration priority |
|---|
<canvas> | role="img", tagName="CANVAS" | canvas.getContext('2d'/'webgl'), width, height | High |
<iframe> | role="iframe", src attribute | frameLocator available | High |
| CAPTCHA | .g-recaptcha, .h-captcha, [data-sitekey], canvas+slider | recaptcha score via API (if configured) | High |
| OTP / SMS | 6-digit input, countdown timer | Check if dev bypass exists | High |
| Shadow DOM | role="generic" with no children | Check shadowRoot via evaluate | Medium |
| Rich text editor | [contenteditable], role="textbox" | innerHTML, getContent() | Medium |
| Video / Audio | role="application" or name contains "video"/"audio" | evaluate checks both <video> and <audio> tags | Medium |
| File upload | <input type="file"> | accept attribute, multiple flag | Medium |
| Drag-and-drop | drag events in JS | Simulate DnD via coordinate clicks | Low |
| Date picker | specific data-testid or class patterns | Click triggers โ evaluate value | Low (skip unless specs mention) |
| Infinite scroll | Dynamic row insertion | Count elements before/after scroll | Low (skip unless specs mention dynamic lists/pagination) |
| WebSocket / SSE | No DOM signal | Check $B console --errors for WS events | Low (check only if app uses real-time features) |
For each detected special element, capture via $B js with targeted DOM queries:
- Canvas:
getContext('webgl2'/'webgl'/'2d'), width, height
- Iframe:
src attribute โ use frameLocator in tests
- CAPTCHA:
.g-recaptcha, .h-captcha, [data-sitekey], canvas+slider detection
- OTP:
input elements with maxLength === 1 or type === 'tel'
- Rich text:
[contenteditable] โ innerHTML, textContent.length
- Video/Audio:
querySelector('video'/'audio') โ paused, duration
- Shadow DOM:
role="generic" with no children โ check shadowRoot
Record findings in app-exploration.md โ Special Elements Detected table.
4.4. Write app-exploration.md
Output: openspec/changes/<name>/specs/playwright/app-exploration.md
"all" mode (no change context): Output to <root>/app-exploration.md instead. The <root>/ version also serves as the INPUT file read by openspec-pw explore (which updates only the Status column).
Key fields per route:
- URL:
${BASE_URL}<path>
- Auth: none / required (storageState:
<path>)
- Ready signal: how to know the page is loaded
- Elements: interactive elements with verified selectors (see 4.3 table)
- Screenshot:
__screenshots__/<slug>.png
After exploration, add route-level notes (redirects, dynamic content โ see 4.5).
4.5. Exploration behavior notes
| Situation | Action |
|---|
| SPA routing (URL changes but page doesn't reload) | Explore via navigation clicks from known routes, not direct URLs |
| Page loads but no interactive elements | Wait longer for SPA hydration |
| Dynamic content (user-specific) | Record structure โ use toContainText or regex, not toHaveText |
Idempotency: If app-exploration.md already exists โ read it, verify routes still match the live app, update only new routes or changed pages.
Route Snapshot Hash -- Skip Unchanged Routes
Before re-exploring, compute a lightweight hash of the app's current state:
- Quick hash (for re-runs): Navigate to
${BASE_URL}/sitemap.xml โ hash the XML content
- If hash unchanged since last exploration: Skip re-exploration entirely -- use cached
app-exploration.md
- If hash changed: Re-explore only changed routes (diff sitemap XML)
- Store hash in
app-knowledge.md โ Exploration State section
## Exploration State
Last explored: 2026-04-12T07:30:00Z
Sitemap hash: sha256:abc123...
Routes count: 20
This prevents redundant exploration on every /opsx:e2e run.
4.6. Update app-knowledge.md
After writing app-exploration.md, extract project-level shared knowledge and append to tests/playwright/app-knowledge.md. Auto-de-duplicate: before adding any row, check if the same key already exists โ if so, skip.
| Section | What to extract | De-duplication key |
|---|
| Architecture | Monolith or separated? Backend port? Restart command? | Section-level: update existing rows instead of adding duplicates |
| Credential Format | Login endpoint, username format (email vs username) | Field-level: Field column (username, password, login endpoint) |
| Common Selector Patterns | New patterns discovered that apply across routes | Element + Selector: skip if same Element + Selector already exists |
| SPA Routing | SPA framework, routing behavior | Section-level: update existing instead of appending |
| Project Conventions | BASE_URL, auth method, multi-user roles | Convention column: skip if same convention already exists |
| Selector Fixes | Healed selectors (see Step 9 Phase 2-7) โ route, old โ new selector, reason, date | Route + Old Selector: skip if same key exists |
| Assertion Fixes | Healed assertions (see Step 9 Phase 2-7) โ test, old โ new assertion, reason, date | Test + Old Assertion: skip if same key exists |
Append only new/changed items โ preserve existing content.
4.7. After exploration
Pass app-exploration.md to:
- Step 5 (Planner): reference real routes, auth states, and elements in test-plan.md
- Step 6 (Generator): use verified selectors instead of inferring
Read tests/playwright/app-knowledge.md as context for cross-change patterns.
5. Generate test plan
"all" mode: skip test-plan generation. No OpenSpec specs โ no test-plan to generate. Still show confirmation below, then proceed to Step 6.
All mode โ brief confirmation before Step 6:
## All Mode: Page Object Discovery
Discovered <N> routes (<M> guest, <K> protected)
Special elements: <element summary>
Ready to generate Page Objects for: <page-name>Page.ts, <page-name>Page.ts, ...
Reply **yes** to proceed, or tell me to exclude routes or adjust strategies.
Change mode โ prerequisite: If openspec/changes/<name>/specs/playwright/app-exploration.md does not exist โ STOP. Run Step 4 (explore application) before generating tests. Without real DOM data from exploration, selectors are guesses and tests will be fragile. All mode: if <root>/app-exploration.md does not exist โ same STOP.
Change mode: Create openspec/changes/<name>/specs/playwright/test-plan.md.
Read inputs: specs, app-exploration.md, app-knowledge.md
Create test cases: functional requirement โ test case, with @role and @auth tags. Reference verified selectors from app-exploration.md.
State mutual exclusion: before creating each test case โ where are the state boundaries? During state transitions, which elements disappear or appear? Mutual exclusion must be asserted explicitly.
If a test case requires page.route() API mocking โ append โ ๏ธ API Mock flag to the test case line in the summary, with reason.
Idempotency: If test-plan.md exists โ read and use, but you MAY supplement missing test cases. "Do not regenerate" means: do not discard existing cases, but you CAN add new ones discovered during Step 4 exploration that weren't in the original spec (e.g., empty states, error paths found during DOM exploration).
โ ๏ธ Human verification โ STOP before generating code.
After creating (or reading existing) test-plan.md, stop and display the test plan summary for user confirmation:
Output format โ show the test plan in markdown directly in the conversation:
## Test Plan Summary: `<change-name>`
**Auth**: required / not required | Roles: ...
### Test Cases
- โ
`<test-name>` โ `<route>`, happy path
- โ
`<test-name>` โ `<route>`, error path: `<error condition>`
### Special Elements
- โ ๏ธ **CAPTCHA** at `<route>` โ strategy: `auth.setup bypass / skip / api-only`
- โ ๏ธ **Canvas/WebGL** at `<route>` โ strategy: screenshot + dimensions
- โ ๏ธ **OTP** at `<route>` โ strategy: test credentials / dev bypass
- โ ๏ธ **Iframe** at `<route>` โ strategy: frameLocator + assert inner content
- โ ๏ธ **Video/Audio** at `<route>` โ strategy: play() + assert !paused
- โ ๏ธ **File Upload** at `<route>` โ strategy: setInputFiles + assert upload
- โ ๏ธ **Drag-and-Drop** at `<route>` โ strategy: dragAndDrop or evaluate events
- โ ๏ธ **WebSocket/SSE** at `<route>` โ strategy: waitForResponse + waitForFunction
### Not Covered
- `<element or scenario not testable>`
Important: Only list special elements that were actually detected in Step 4. Do not pre-populate with all possible types. If no special elements were found โ omit the Special Elements section entirely.
Then ask: "Does this coverage match your intent? Reply yes to proceed, or tell me what to add/change."
If the user requests changes โ update test-plan.md โ re-display summary โ re-confirm โ proceed.
6. Generate (Generator role)
"all" mode: Build and expand Page Objects for future Change tests.
Prerequisite (change mode only): If openspec/changes/<name>/specs/playwright/app-exploration.md does not exist โ STOP. Run Step 4 first. For all mode, exploration is embedded dynamically in Step 6 โ no pre-existing app-exploration.md is required.
Page Object pattern โ read before writing any page file:
export class LoginPage extends BasePage {
get usernameInput() { return this.byLabel('็จๆทๅ'); }
get submitBtn() { return this.byRole('button', { name: '็ปๅฝ' }); }
constructor(page: Page) { super(page); }
async login(user: string, pass: string) {
await this.goto('/login');
await this.fillAndVerify(this.usernameInput, user);
await this.click(this.submitBtn);
}
}
test('login', async ({ page }) => {
await page.getByLabel('็จๆทๅ').fill('user');
});
Decision table โ Page Object file handling:
| Situation | Action |
|---|
pages/<Route>Page.ts does not exist | Create from LoginPage pattern |
| File exists with some getters | Extend โ add missing, preserve existing |
| File exists but uses inline locators | Rewrite with Page Object pattern, keep selector strings |
| Route removed from app | Remove corresponding Page Object file |
File naming: pages/<Route>Page.ts โ use kebab-case route โ PascalCase. /login โ LoginPage.ts, /user-profile โ UserProfilePage.ts.
For each discovered route:
- Read existing
pages/<Route>Page.ts (if any โ incremental, not overwrite)
- Navigate to route with correct auth state
- browser_snapshot to extract interactive elements (see 4.3 table)
- Write or update
pages/<Route>Page.ts โ extend with newly discovered elements
- Also write
tests/playwright/app-all.spec.ts โ smoke test. Minimum standard: verify at least one heading or key interactive element is visible โ not just "no crash". If the page loads but shows an empty shell or an error, the test must fail.
Output priority: Page Objects (pages/*.ts) are the primary asset. Smoke test is secondary. Existing Page Objects are never overwritten โ only extended.
Change mode โ tests/playwright/changes/<name>/<name>.spec.ts (functional):
- Read: test-plan.md, app-exploration.md, app-knowledge.md, seed.spec.ts
- For each test case: verify selectors in real browser, then write Playwright code
Per-assertion UI check (before writing each assertion):
Is this assertion about a visible UI result?
โ Yes โ MUST use: expect(locator) with page selector
โ No โ Is this a precondition or unreachable HTTP error?
โ Yes โ page.request is acceptable (record reason)
โ No โ This is a bug โ rewrite with UI selector
Never use page.request for assertions the user can see on screen. If you wrote page.request.get() for a visible result โ rewrite with expect(locator) from the browser snapshot.
Selector verification (change mode):
- Navigate to route with correct auth state
- browser_snapshot to confirm page loaded
- For each selector: verify from current snapshot (see 4.3 table for priority)
- Write test code with verified selectors
- If selector unverifiable โ note for Healer (Step 9)
Selector Caching โ reuse Step 4 exploration results:
After Step 4, verified selectors are stored in app-exploration.md. Before navigating to verify a selector in Step 6, check if the route already has verified selectors from Step 4.
const cachedSelectors = appExploration.routes.find(r => r.path === routePath)?.elements
if (cachedSelectors && cachedSelectors.length > 0) {
const selector = getFirstPrioritySelector(cachedSelectors)
if (selector.stability !== 'Fragile') {
return selector
}
}
Benefit: For a 50-test case suite, this saves 30-50 redundant navigations (~2-5 minutes).
File to read: The selector caching uses data already stored in Step 4.4's app-exploration.md output โ no new file needed.
Test coverage โ empty states: For list/detail pages, explore the empty state. If the app shows a "no data" UI when the list is empty, generate a test to verify it. Empty states are often missing from specs but are real user paths.
Test coverage โ special elements: Check app-exploration.md โ Special Elements Detected table. For each special element, generate tests using the following strategies:
- Canvas: screenshot + boundingBox โ dimensions > 0, or 2D pixel verification
- WebGL: screenshot only (no pixel comparison โ rendering varies)
- Iframe:
frameLocator + assert inner content visible
- Rich text:
contenteditable โ type + textContent assertion
- Video/Audio:
play() โ assert !paused
- CAPTCHA/OTP/File upload/Drag-drop: See AI-Opaque Elements section in templates
Test coverage โ AI-opaque elements: For CAPTCHA, OTP, slider CAPTCHA, file upload, and drag-drop โ elements that Playwright cannot reliably automate:
- Mark the element in
app-exploration.md โ Special Elements Detected table with type and automation strategy
- Generate the test using the appropriate strategy:
- CAPTCHA: Bypass via
auth.setup.ts storageState, or skip with test.skip(), or verify via API
- OTP: Use pre-verified test credentials (
E2E_OTP_CODE env var), or development bypass flag
- File upload: Use
page.setInputFiles() with fixture files
- Drag-drop: Use
page.dragAndDrop() or page.evaluate() with custom event dispatching
- If the element is truly non-automatable, write
test.skip() with a comment explaining why, and mark with /handoff for manual testing
Test coverage โ performance: Generate a Core Web Vitals test only if the OpenSpec spec or app-exploration.md specifies explicit performance targets (e.g., "LCP must be under 2s"). If no business target is defined, skip performance testing โ hard-coded thresholds (lcp < 2500ms) produce false passes and add noise.
await canvas.screenshot()
await expect(canvas).toHaveScreenshot()
const box = await canvas.boundingBox();
expect(box.width).toBeGreaterThan(0);
Output format:
- Follow
seed.spec.ts structure
- Use
test.describe(...) for grouping
- Each test:
test('ๆ่ฟฐๆงๅ็งฐ', async ({ page }) => { ... })
- Prefer
data-testid selectors (see 4.3 table)
6.1. Use BasePage for shared navigation and selectors
Read tests/playwright/pages/BasePage.ts for shared utilities:
goto(path) โ navigation with configurable waitUntilbyRole(role, opts), byLabel(label), byPlaceholder(text), byText(text), byTestId(id) โ selector helpers in priority order (semantic โ form โ fallback)click(locator), fill(locator, value), fillAndVerify(locator, value) โ safe interactions; use fillAndVerify when the next action depends on the value being committedwaitForToast(text?), waitForLoad(spinnerSelector?) โ wait utilitiesreload() โ page reload with hydration
AppPage pattern โ extend BasePage for page-specific selectors:
import { BasePage } from './BasePage';
import type { Page } from '@playwright/test';
export class LoginPage extends BasePage {
get usernameInput() { return this.byLabel('็จๆทๅ'); }
get passwordInput() { return this.byLabel('ๅฏ็ '); }
get submitBtn() { return this.byRole('button', { name: '็ปๅฝ' }); }
constructor(page: Page) { super(page); }
async login(user: string, pass: string) {
await this.goto('/login');
await this.fillAndVerify(this.usernameInput, user);
await this.fillAndVerify(this.passwordInput, pass);
await this.click(this.submitBtn);
}
}
import { LoginPage } from '../pages/LoginPage';
test('user can login', async ({ page }) => {
const loginPage = new LoginPage(page);
await loginPage.login('user@example.com', 'password123');
await loginPage.expectURL(/dashboard/);
});
If a shared page object doesn't exist yet: define it inline in the spec AND write it to tests/playwright/pages/<PageName>.ts so future tests can reuse it.
6.2. Selector patterns
| Prefer (robust) | Avoid (fragile) |
|---|
getByRole, getByTestId, getByLabel | CSS class ('.notification-bell'), CSS ID ('#avatarBtn') |
waitForSelector(targetElement) | hardcoded 200ms / 500ms delays |
See above for Page Object pattern, LoginPage example, and BasePage utilities.
If the file exists โ diff against test-plan, add only missing test cases.
7. Configure auth (if required)
- API login: Generate
auth.setup.ts using E2E_USERNAME/E2E_PASSWORD + POST to login endpoint
- UI login: Generate
auth.setup.ts using browser form fill. Update selectors to match your login page
- Multi-user: Separate
storageState paths per role
Credential format guidance:
- If the app uses email for login โ use
CHANGE_ME@example.com
- If the app uses username (alphanumeric + underscore) โ use
test_user_001 (more universal)
- Check existing test files or login page to determine the format
- Always set credentials via environment variables โ never hardcode
Prompt user:
Auth required. To set up:
1. Customize tests/playwright/credentials.yaml
2. Export: export E2E_USERNAME=xxx E2E_PASSWORD=yyy
3. Run auth: npx playwright test --project=setup
4. Then run tests: openspec-pw run <name> # skips to Step 9 directly (artifacts are reused)
Idempotency: If auth.setup.ts already exists โ verify format, update only if stale.
Post-auth re-exploration: If Step 4 skipped protected routes due to missing auth, re-run exploration for those routes now that auth is configured. Navigate to each protected route with auth context โ $B snapshot โ update app-exploration.md. Selectors verified now are better than guesses used during test generation.
8. Configure playwright.config.ts
Output: playwright.config.ts (project root; or tests/playwright/playwright.config.ts if config already exists there)
If missing โ generate a minimal playwright.config.ts with webServer, projects, and reporters.
Auto-detect BASE_URL (in priority order):
process.env.BASE_URL if already settests/playwright/seed.spec.ts โ extract BASE_URL value- Read
vite.config.ts (or vite.config.js) โ extract server.port + infer protocol (https if server.https, else http)
- Read
package.json โ scripts.dev or scripts.start โ extract port from --port flag
- Fallback:
http://localhost:3000
Auto-detect dev command:
package.json โ scripts in order: dev โ start โ serve โ preview โ npm run dev
If playwright.config.ts exists โ READ first, preserve ALL existing fields, add only missing webServer block.
9. Execute tests
openspec-pw run <name> [--project <role>] [--headed] [--update-snapshots]
The CLI handles: server lifecycle, port mismatch, report generation.
If tests fail โ use Playwright MCP tools to inspect UI, fix selectors, re-run.
Browser visibility: The Healer uses browser MCP tools (snapshot, screenshot, console messages) to inspect failures โ no need for --headed. If you want to watch the browser yourself during debugging, add --headed: openspec-pw run <name> --headed.
Healer MCP tools (in order of use):
| Tool | Purpose |
|---|
browser_navigate | Go to the failing test's page |
browser_snapshot | Get page structure to find equivalent selectors |
browser_console_messages | Diagnose JS errors that may cause failures |
browser_network_requests | Diagnose backend/API failures (4xx/5xx) |
browser_take_screenshot | Visually compare before/after fixes |
browser_run_code | Execute custom fix logic (optional) |
Before Phase 1 โ check accumulated knowledge:
Read tests/playwright/app-knowledge.md โ Selector Fixes table. If the failing test's selector or route matches a known fix, use it directly (skip Phase 2. If partial match โ use as the top candidate in Phase 2-5a). If file does not exist โ skip.
Healer โ Phase 1: Triage
When a test fails, classify before attempting repair.
Batch Failure Detection โ run this FIRST when multiple tests fail:
Collect ALL failing test names + their failure reasons.
Group by: same route + same action + same Playwright error type (e.g., both "element not found" with the **same** root cause confirmed by browser_console_messages showing no JS error and browser_network_requests showing no 4xx/5xx). If root cause differs (e.g., element A missing vs element B in different section), treat as **separate** groups.
If โฅ2 tests fall into the same group:
โ Pause individual healing
โ Navigate to that route + perform the action manually
โ Check browser_console_messages + browser_network_requests
โ If console error or 4xx/5xx present:
โ This is an App Bug (backend/API change), NOT Test Bugs
โ Classify all tests in this group as App Bug
โ Skip all โ record 1 App Bug in registry (not N bugs)
โ Skip the rest of individual Triage for this group
โ If timeout errors:
โ Retry one isolated: `npx playwright test --grep "<first-test-name>"` [--project <role>]
โ If isolated passes โ **RAFT** (shared state coupling) โ skip all โ note RAFT in report
โ If isolated still fails โ individual **Flaky** (each test independently flaky) โ proceed with individual Triage for this group
โ If no console/network error and no timeout, but all still fail:
โ Likely a shared state issue โ **RAFT** โ skip all โ note RAFT in report
Proceed with individual Triage only for tests NOT in a batch failure group.
After Batch Detection, individual Triage:
| Failure Type | Signal | Classification | Action |
|---|
| Network/Backend | net::ERR, 4xx/5xx in console/network | App Bug | test.skip() + record in app-bug-registry.md |
| JS Runtime Error | Console error (non-network) | App Bug | test.skip() + record in app-bug-registry.md |
| Auth Expired | Redirected to login mid-test | Flaky | Re-run auth.setup โ re-run |
| Redirect Loop | ERR_TOO_MANY_REDIRECTS, URL keeps changing on each snapshot | App Bug | Check auth first (see Step 4.2 loop detection). If auth is the cause โ re-run auth. Otherwise โ test.skip() + App Bug Registry |
| Page Refresh Loop | Excessive console errors (>10) after navigation, page unstable | App Bug | test.skip() + record in App Bug Registry |
| Selector Not Found | Element not found | Test Bug | โ Phase 2 Healer |
| Assertion Mismatch | Wrong content/value | Ambiguous | โ Phase 2 Healer |
| Timeout | waitFor/evaluate timeout | Flaky | Retry isolated: npx playwright test --grep "<test-name>" (1ร, not counted in heal attempts). If it passes isolated but fails in suite โ RAFT. If it consistently times out โ check framework: React 19 / Next.js App Router: add page.waitForLoadState('networkidle'). Vue/Angular/React 18 / Plain JS / jQuery: use waitForSelector(targetElement) instead of timeout tuning. |
| Same test fails in suite, passes isolated | โ | RAFT | test.skip() in suite, note RAFT in report |
- App Bug โ skip immediately (no healing needed) โ record in App Bug Registry
- Flaky โ retry once isolated
- Test Bug / Ambiguous โ Phase 2
App Bug Registry
For every App Bug classified in Phase 1, record it in openspec/reports/app-bug-registry.md (create if missing):
# App Bug Registry
<!-- Auto-generated. Do not edit manually. -->
## Active App Bugs
| # | Test | Route | Signal | First Detected | Status |
|---|------|-------|--------|---------------|--------|
| 1 | test-name | /route | net::ERR_CONNECTION_REFUSED | 2026-04-09 | open |
Update rules:
- New App Bug: Append new row, increment
#
- App Bug re-run and now passes: Keep row, change status to
resolved + add Resolved column with date
- Keep all rows (never delete) โ the resolved count is the signal that bugs are being fixed
Why this matters: test.skip() hides App Bugs from the pass/fail count. Without an explicit registry, "all tests passed" is a false positive when App Bugs exist. The registry makes the invisible visible.
Global attempt guard: Each test has an independent heal counter (max 3 per test). If the same test enters Phase 2 more than once and reaches the cap each time โ treat as "consecutive escalation without progress" โ Phase 3 immediately.
Type โ Blame: "Test Bug" means the assertion or selector is wrong โ it does NOT mean "blame the test author." The test was generated from the spec. Root cause may be spec ambiguity, specโtest generation error, or appโspec deviation. Only a human can determine blame.
Healer โ Phase 2: Repair
After Triage classifies failure as "Test Bug" or "Ambiguous":
Phase 2-0 โ Batch diagnosis (no Playwright, fast):
For ALL Phase 2 failures in this run, do this simultaneously:
- Read the failing test spec file(s) to understand what each test verifies
- Read
app-knowledge.md for any previous fixes or selector patterns
- Read
app-knowledge.md โ Common Selector Patterns for project conventions
- For each test, output:
TEST: <test-name>
ROUTE: <route>
ASSERTION: "<what the test expects>"
EXPECTED_BEHAVIOR: <from spec, one line>
KNOWN_FIX: <yes/no โ from app-knowledge.md Selector Fixes table>
- Classify each test:
ready-to-fix โ known fix exists, or selector pattern is clearneeds-assertion-fix โ assertion itself needs changing (typo, wrong expected value)needs-phase3 โ ambiguous, no clear fix pathneeds-more-diagnosis โ need to see actual page to determine
Skip remaining steps for needs-phase3 tests โ go directly to Phase 3.
For needs-assertion-fix tests โ go to Phase 2-1 (navigate + snapshot โ assertion fix).
For ready-to-fix tests โ go to Phase 2-5 (selector repair).
For needs-more-diagnosis tests โ go to Phase 2-1 (navigate + snapshot) for each individually.
Phase 2-1 (for needs-assertion-fix or needs-more-diagnosis): Navigate to the failing page โ browser_snapshot โ EXPLICIT COMPARISON:
ASSERTION: "<what the test expects>"
ACTUAL: "<what the snapshot shows>"
MATCH: <yes/no>
- If MATCH=yes โ apply the assertion fix โ go to Phase 2-6
- If MATCH=no:
- Safe to fix without Phase 3 โ apply the fix โ go to Phase 2-6
- "Never fix without Phase 3" conditions met โ Phase 3 immediately
Assertion modification guard โ never skip Phase 3 unless ALL conditions are met:
- The test has never passed with the current assertion (newly generated test, or this is the first time this assertion fails)
- The ACTUAL value is verifiably from a different spec section (e.g., this test was in the wrong describe block)
- You can point to the specific line in the spec that defines the ACTUAL behavior
If ANY condition is uncertain โ Phase 3 immediately. Do NOT modify the assertion.
Safe to fix without Phase 3:
- Typo in assertion (e.g., "Subm it" vs "Submit" in the expected text)
- Selector was correct but the element was moved to a different location (same text, different selector)
- Explicit spec drift confirmed by reading the spec (e.g., spec says "button says Submit" but test says "button says Submit Form")
Never fix without Phase 3:
- App behavior changed after an action (e.g., "after clicking submit, balance should decrease" โ ACTUAL shows no change โ Phase 3, could be optimistic update bug, backend failure, or spec mismatch)
- Data values differ (e.g., expected "ยฅ1000" but got "ยฅ999" โ Phase 3, could be rounding, discount, or calculation bug)
- Missing elements after interaction (e.g., "after creating order, success message should appear" โ no message โ Phase 3)
Phase 2-5 (for ready-to-fix with selector issue): Generate candidate list from snapshot:
Phase 2-5a. Extract candidates โ identify the target element from the failing test's assertion.
If KNOWN_FIX=yes from Phase 2-0 diagnosis โ read the New Selector from app-knowledge.md โ Selector Fixes table (same Route + Old Selector row) โ apply directly โ go to Phase 2-6 (no need to re-generate candidates from snapshot).
Otherwise โ list all selectors for the target element from the snapshot, and check app-knowledge.md โ Common Selector Patterns for project-specific conventions.
Target: <element from failing assertion, e.g. "button with text 'Submit'">
Candidates (stable โ fragile):
- getByRole(button, { name: 'Submit' }) โ Stable (semantic)
- getByText('Submit') โ Fair (unique text)
- getByLabel('Email') โ Fair (form fields)
- locator('#submit') โ Fair (id attribute)
- locator('.btn-primary') โ Fragile (style class) โ upgrade if listed in Common Selector Patterns
- locator('button:nth-child(3)') โ Fragile (DOM order)
Stability: getByRole > getByText/getByLabel > locator('#id') > locator('.class') > locator('nth-child'). Upgrade stability if app-knowledge.md โ Common Selector Patterns explicitly lists the selector as preferred for this project.
Phase 2-5b. Select top candidate โ pick the highest-stability candidate that matches the target. Output: SELECTED: <selector> โ reason: <why this one>. Then โ go to Phase 2-6.
Phase 2-6 โ Fix and verify incrementally (per test):
- For each test needing a fix, apply the fix to the
.spec.ts file
- Run:
npx playwright test --grep "<test-name>" to verify that specific test
- If passed โ Phase 2-7 (log to app-knowledge.md), then move to next test
- If failed โ analyze the new failure type:
- Assertion-related (wrong value/content) โ return to Phase 2-1 for that test
- Selector-related (element not found, wrong element) โ return to Phase 2-5 for that test (re-diagnose selector, not Phase 2-1)
- Timeout-related โ treat as Flaky โ retry isolated once:
npx playwright test --grep "<test-name>" [--project ]
- If isolated passes โ Phase 2-7 (healed), then move to next test
- If isolated still fails โ apply framework fix (React 19 / Next.js:
page.waitForLoadState('networkidle'); Vue/Angular/React 18 / Plain JS: waitForSelector(targetElement)) โ retry โ if passes โ Phase 2-7 (healed) โ move to next test; if still fails โ test.skip() in .spec.ts + note in report โ move to next test (not counted in heal attempts)
The heal counter does NOT reset for any path.
- Repeat until all Phase 2 tests are healed.
Why --grep instead of --only-changed: Healer fixing is incremental โ you fix one test, verify it, adjust if needed, then move to the next. --only-changed is file-level: it runs ALL tests in .spec.ts files that have uncommitted source changes, not individual tests. This wastes time in an iterative fix cycle (re-running already-passed tests in the same file). Use --grep for targeted verification. Reserve --only-changed for the pre-commit guard.
Pre-commit guard โ --only-changed (after all Phase 2 tests are healed): npx playwright test --only-changed runs all tests in changed .spec.ts files (comparing against HEAD by default, or --only-changed=main against a branch). Playwright also analyzes source file dependencies โ if src/components/Button.ts changed, any .spec.ts that imports it will run. CI still runs the full suite as complete regression.
Phase 2-7 โ If healed โ append to app-knowledge.md (auto-de-duplicate):
Selector Fixes โ Before appending, read the existing Selector Fixes table. Skip if the same Route + Old Selector key already has a row (same route + same old selector = duplicate). If the Route + Old Selector exists but with a different New Selector โ this is a new symptom of the same root cause โ add a new row (different date, updated new selector).
Assertion Fixes โ Before appending, read the existing Assertion Fixes table. Skip if the same Test + Old Assertion key already has a row. If the same test had a previous fix with a different old โ new โ add a new row (the assertion has drifted further).
If a new row is added, update the Last updated timestamp in the file header.
This log feeds the Auto-Heal Log in the Phase 10 report.
Element Missing handling (when browser_snapshot shows element not found):
| Situation | Check | Action |
|---|
| JS error in console after action | browser_console_messages | App Bug โ Phase 1 โ App Bug classification |
| Auth redirected mid-action | URL changed to /login | Flaky โ re-run with fresh auth |
| SPA route didn't update | URL is correct but element missing | Wait for SPA hydration โ page.waitForLoadState('networkidle') or waitForSelector(target) |
| Element genuinely missing | None of the above | Test Bug โ find alternative selector or Phase 3 if no equivalent exists |
Healer โ Phase 3: Escalate
When Phase 2 tried โฅ3 heals without success, OR ASSERTION vs ACTUAL comparison is ambiguous:
STOP and output:
E2E Test Failed โ Human Decision Required
Test: <test-name>
Failure: <type>
Assertion: "<what test expects>"
Actual: "<what app shows>"
This failure could be:
1. App does not match the spec โ **app bug**
2. Test was generated from ambiguous/incorrect spec โ **spec issue**
3. Spec itself is outdated (app was updated) โ **spec drift**
Please decide:
(a) Fix the app to match the spec
(b) Update the spec to match the app
(c) Update the test assertion
(d) Skip this test with test.skip() until resolved
Wait for user input before proceeding.
Decision tree โ follow the path based on user's choice:
| Choice | What to do | After fix, do this |
|---|
| (a) Fix the app to match the spec | Fix the app code | Re-run: openspec-pw run <change-name> to verify fix |
| (b) Update the spec to match the app | Edit the spec file | Then update the test assertion (โ option c), or regenerate the affected part of the test |
| (c) Update the test assertion | Fix the assertion in tests/playwright/changes/<name>/<name>.spec.ts | Re-run: openspec-pw run <change-name> to verify |
(d) Skip with test.skip() | Add test.skip() to the test | Note in app-knowledge.md โ Assertion Fixes with reason "human escalation โ skipped pending resolution" |
Stuck in escalation loop (see also: "Global attempt guard" above โ tracked per test, independent heal counter): If 3 consecutive Phase 3 escalations for the same test result in no progress, STOP and ask: "This test has been escalated 3 times without resolution. Are you sure the root cause is still the same, or has something changed?"
After the issue is resolved, re-run tests:
openspec-pw run <change-name>
/opsx:e2e <change-name> re-runs the full 10-step workflow โ unnecessary after Phase 3. The test file and auth context are already correct. Use openspec-pw run to verify fixes directly.
10. False Pass Detection + RAFT Detection + App Bug Accumulation + Report results
Run after test suite completes (even if all pass).
False Pass patterns (test passed but shouldn't have):
- Conditional visibility:
if (locator.isVisible().catch(() => false)) โ if test passes, locator may not exist
- Too fast: < 200ms for a complex flow is suspicious
- No fresh auth context: Protected routes without
browser.newContext()
RAFT detection (Resource-Affected Flaky Test):
- Full suite: test fails โ run isolated:
npx playwright test --grep "<test-name>" [--project ] โ if it passes in isolation but fails in suite โ RAFT
- This is NOT a test bug or app bug. Mark as RAFT, add
test.skip() in suite, note in report
- RAFTs are infrastructure coupling issues (CPU/memory/I/O contention), not fixable by changing test or app
App Bug Accumulation Detection (most critical):
- Read
openspec/reports/app-bug-registry.md
- Count
open status rows
- If โฅ 3 active App Bugs โ add "โ ๏ธ App Bug accumulation: N bugs unresolved" to report summary. Do NOT suppress or hide this warning. The test suite may report "all passed" (skipped โ failed), but N broken features is not a passing system.
- Check for App Bugs that became "resolved" (passing on re-run) โ this is the signal that bugs were fixed. Update
app-bug-registry.md accordingly, and remove test.skip() from those tests so they re-enter the regression suite.
Report results โ compile from these sources (the auto-generated playwright-e2e-<name>-<timestamp>.md provides the Summary table; populate the sections below manually from Phase 1โ3 output):
- Summary table with failure type breakdown (App Bugs, Test Bugs/healed, Flaky-RAFT, Human Escalations) โ from
openspec/reports/playwright-e2e-<name>-<timestamp>.md
- App Bug Summary: Table of all active App Bugs โ test name, route, signal, first detected. If โฅ 3 active โ display "โ ๏ธ App Bug accumulation warning" prominently. โ from
openspec/reports/app-bug-registry.md
- Conditional "All Pass" conclusion:
- โ
"All tests passed" โ only if 0 active App Bugs and 0 skipped tests
- โ ๏ธ "All tests passed (N skipped)" โ if skipped tests exist but no active App Bugs
- โ ๏ธ "All tests passed (N skipped, M App Bugs unresolved)" โ if active App Bugs exist
- Failure Classification table: (test, type, action, healed?) โ compile from Phase 1 Triage + Phase 2-0 classification output
- Auto-heal log: (assertion vs actual comparison, fix applied, result) โ compile from Phase 2-1/2-5 console output +
app-knowledge.md Selector Fixes table
- RAFT Summary (if any): โ compile from RAFT detection notes during Phase 1
- Human Escalations (if any, with user decision): โ from Phase 3 decision output
- Recommendations with
file:line references
Update tasks.md:
- If 0 active App Bugs โ find E2E-related items, append
โ
Verified via Playwright E2E (<timestamp>).
- If active App Bugs exist โ do not mark as verified. Append instead:
โ ๏ธ App Bug blocked: <bug summary> (<timestamp>).
Graceful Degradation
| Scenario | Classification | Action | Workflow Status |
|---|
| No specs / app-exploration.md missing (change mode) | Blocker | STOP โ check path: openspec/changes/<name>/specs/playwright/app-exploration.md (change) or <root>/app-exploration.md (all mode) | Stops entirely |
| JS errors or HTTP 5xx during exploration | Blocker | STOP โ user fixes app โ re-run /opsx:e2e <name> from Step 4 | Stops entirely |
| Redirect loop / page refresh loop during exploration | App Bug | Skip route โ record in App Bug Registry โ re-run after fix | Continues (other routes) |
| File already exists (app-exploration, test-plan, app-all.spec.ts, Page Objects) | Idempotency | Read and use โ never regenerate | Continues |
| Test fails (network/backend) | App Bug | test.skip() + record in app-bug-registry.md | Continues (workflow level โ no STOP) |
| Test fails (selector/assertion) | Test Bug/Ambiguous | Healer Phase 1โ2 (โค3 attempts) | Continues |
| RAFT detected (suite fail, isolated pass) | Flaky | test.skip() in suite, note RAFT in report | Continues |
| Phase 3 escalation | Human needed | STOP + present 4 options โ wait for user decision | Stops entirely |
| False pass detected | Coverage Gap | Add "โ ๏ธ Coverage Gap" to report | Continues |
| App Bug skip accumulation (โฅ3 active App Bugs) | Warning | Add "โ ๏ธ App Bug accumulation: N bugs unresolved" to report. Do not suppress. | Continues |
"STOP" = workflow halts and waits for user input (true stop).
"Continues" = workflow proceeds to next step (test-level skip, no user wait).
Guardrails
Decision table:
| Rule | Why |
|---|
| Read specs as source of truth | Generated tests must match requirements |
| Step 4 before Step 6 | Real DOM data โ accurate selectors |
| Never contradict specs | E2E validates implementation, not design |
| Cap heal at 3 attempts | Prevents infinite loops |
| Write runnable code, not TODOs | Placeholders fail CI |
Files you can write to:
tests/playwright/, openspec/changes/<name>/specs/playwright/, openspec/reports/, playwright.config.ts, auth.setup.ts
tests/playwright/ โ spec files, Page Objects, auth, credentials, app-knowledge.md
openspec/changes/<name>/specs/playwright/ โ app-exploration.md, test-plan.md (change mode)
<root>/ โ app-exploration.md (change mode: INPUT for openspec-pw explore; all mode: detailed output)
openspec/reports/ โ test reports, app-bug-registry.md
Never write to: any other directory
