diagnose-pw-failure

name	diagnose-pw-failure
description	Diagnose a Playwright test failure as a product bug by analyzing error output, screenshots, traces, and server logs. Produces a structured bug report with expected vs actual behavior, reproduction steps, and likely root cause. Use when a Playwright test keeps failing after spec fixes, "is this a product bug", "diagnose this test failure", or "the test is correct but the app is broken".

Diagnose Playwright Failure as Product Bug

When to Use

Use this skill when:

• A spec has been fixed 2-3 times and the same assertion keeps failing with the same received value
• The spec's selectors and waits are correct but the app renders wrong content
• The server returns unexpected status codes (500, 403) on valid requests
• The write-and-verify-pw-test or fix-pw-spec skill has exhausted its retry budget

Do not use if the error is clearly a test code issue (import error, wrong selector syntax, TypeScript compilation error). Use fix-pw-spec instead.

Step 1 — Collect Evidence

Gather all available evidence:

a) Playwright error output

Read the terminal output from the failed test run. Note:

• The exact assertion that failed
• Expected vs received values
• Which test step failed (line number in the spec)

b) Screenshots

Check app/client/playwright/results/ for failure screenshots:

ls -la app/client/playwright/results/

If screenshots exist, read them (the Read tool supports images). They show exactly what the browser rendered at failure time.

c) Trace (if available)

Traces are at app/client/playwright/results/<test-path>/trace.zip. Note their location for the bug report — they can be viewed with npx playwright show-trace <path>.

d) Network responses

If the spec uses waitForResponse, check if the API response was captured in the error output. Common patterns:

• API returned 200 but with wrong data → backend logic bug
• API returned 500 → server crash
• API returned 403 → permission/auth regression
• API never responded (timeout) → endpoint broken or renamed

e) Server-side code (optional, for deeper diagnosis)

If the failure involves an API call, trace the server code:

1. Identify the API endpoint from the spec (e.g., API.actionsExecute → /api/v1/actions/execute)
2. Find the controller: grep app/server/ for @PostMapping("/api/v1/actions/execute") or similar
3. Read the service method to understand what could go wrong
4. Check if recent commits changed this code path

Step 2 — Classify the Bug

Category	Signals	Severity
UI rendering	Wrong text, missing element, broken layout (screenshot shows it)	Medium
Data regression	API returns correct status but wrong payload	High
Server error	500 response, stack trace in logs	Critical
Auth/permission	401/403 on previously working endpoint	High
Feature flag	Feature works with flag on but not off (or vice versa)	Medium
Deployment issue	ECONNREFUSED, DNS failure, unhealthy containers	Blocker
Race condition	Intermittent — passes sometimes, fails others	Medium (flaky)

Step 3 — Verify It's Not a Spec Bug

Before declaring "product bug", do a sanity check:

1. Manual verification: Does the spec's assertion make sense? Re-read the test name and expected behavior.
2. Check the deployment manually: Navigate to PLAYWRIGHT_BASE_URL in your analysis and verify the page actually shows what the test expects.
3. Check if the feature exists on this deployment: The deployment might be on an older version that doesn't have the feature yet.
4. Check feature flags: If the feature is behind a flag, verify the flag is enabled on the deployment (check PW_FLAG_OVERRIDES or query /api/v1/users/features).

Step 4 — Produce Bug Report

Output a structured diagnosis:

## Playwright Failure Diagnosis: PRODUCT BUG

**Spec**: playwright/tests/sanity/widgets/table-filter.spec.ts
**Test**: "filters table by country"
**Deployment**: https://my-dp.appsmith.com
**Category**: Data regression

### Expected behavior
Filtering the table by Country "starts with Ba" should show rows including "Bangladesh".

### Actual behavior
Filter returns 0 rows. The table is empty after applying the filter.

### Evidence
- **Assertion**: `expect(table.cell(2, 0)).toContainText("Bangladesh")` — timed out, cell doesn't exist
- **Screenshot**: playwright/results/sanity-widgets-table-filter/test-failed-1.png
  - Shows table with "No data" message after filter is applied
- **API response**: GET /api/v1/actions/execute returned 200 with `{ data: [] }`
- **Spec verified correct**: Selector targets the right table, filter UI interaction works (filter chip appears)

### Likely root cause
The execute API returns empty results for the MySQL "starts with" filter. Possible causes:
- Query generation bug in the server's filter-to-SQL translation
- Datasource connection issue (DATASOURCE_HOST may be unreachable from this deployment)

### Reproduction steps
1. Open the app in deployed mode
2. Click "Add Filter" on the data_table widget
3. Set Column: Country, Condition: starts with, Value: Ba
4. Observe: table shows "No data" instead of filtered results

### Suggested investigation
- Check server logs for the execute query
- Verify DATASOURCE_HOST is reachable from the deployment
- Test the same filter on dev.appsmith.com to compare

When Diagnosis Is Inconclusive

If you can't determine whether it's a spec bug or product bug:

## Playwright Failure Diagnosis: INCONCLUSIVE

**Spec**: <path>
**Test**: <name>

### What we know
- [facts from error output]

### What we don't know
- [ambiguities]

### Recommended next steps
1. Run the test with `--debug` flag for step-by-step execution
2. Check server logs on the deployment
3. Try reproducing manually in the browser