skytest-fix

name

skytest-fix

description

Diagnose and fix failed SkyTest test cases. Analyzes run reports, events, screenshots, and error traces to classify failures into a systematic taxonomy — then proposes targeted fixes and applies them via MCP after user confirmation. Default stabilization strategy: convert flaky ai-action interaction steps (clicks, fills, dropdowns, datepickers) to playwright-code with verified selectors. Use when a test run has failed and you need to investigate and fix the test case.

SkyTest Fix Skill

Diagnose failed test runs and fix test cases systematically.

Non-Negotiable Rules

• Never apply fixes without user confirmation. Present diagnosis and proposed fix first.
• Never fabricate Playwright selectors. Get them from a connected browser tool (Playwright MCP / Chrome DevTools MCP), screenshots showing exact labels, an existing passing step, or the user. Role/label locators from exact visible labels are acceptable for standard controls — verify live when a browser tool is connected.
• Verify fixes with browser tools when available — navigate to the failing page state, test the selector/interaction there, and only then propose it. Do this silently during investigation.
• Trace the root cause before proposing. A step often fails because a previous step left the page in the wrong state. Read the full event log and screenshots, not just the error message.
• One fix at a time. Fix the first failing step, re-run, then reassess. Don't speculate about downstream failures.
• For anything unclear, ask the user. Don't assume page structure or expected behavior.

Playwright-First Stabilization

Validated in practice: deterministic interactions (clicking, filling, selecting dropdowns/datepickers/checkboxes) run far more reliably as playwright-code than as ai-action. When an ai-action interaction step fails or flakes:

• Converting it to playwright-code is the default fix, not a last resort — provided selector confidence is met (verified live, exact label known, or user-supplied).
• When converting one step, scan the rest of the test case for other ai-action interaction steps on the same screens. Propose converting them in the same update — it prevents the next flake and saves re-run cycles. Present as one batch for confirmation.
• Keep ai-action for visual/fuzzy verification, scrolling, and screens you cannot capture.

Input

Accepts any of: run ID / test case ID, pasted run report, screenshots, or "test X fails on step 3". If only an ID is given, fetch everything first: get_test_run with include: ["events", "artifacts"] AND get_test_case (steps, variables, configs, last 5 runs).

A run is one member of a run session (a login flow runs as a prefix before the test; a test group runs many cases as one session). get_test_run returns runSessionId, kind, sessionPosition, a cancellationReasonCode, and the rolled-up session. When the run belongs to a session, call get_run_session(runSessionId) to see every member's status — the failure that needs fixing is often in a different member (the login flow) than the one the user pointed at. Classify cancellation-by-upstream-failure (F10) before anything else.

Failure Taxonomy

Classify every failure; the category drives the fix.

F1: AI Model Surface Error — investigate deeper

Signals: "failed to call AI model service", "empty content", "model timeout", "rate limit". These are wrappers, not root causes. Check in order: (1) unresolved {{VARIABLE}} → F7; (2) page in wrong state from previous step → F5; (3) element below viewport / not loaded → F2; (4) vague step description → rewrite, or convert to playwright-code; (5) small icon / complex widget → F2/F3. Only if all check out → transient; suggest re-run.

F2: Element Not Found

Signals: "element not found", repeated locate cycles, locate-phase timeout. Fixes: below viewport → add scroll step; not loaded → add verification gate; label mismatch → fix text to match actual UI; iframe/shadow DOM → playwright-code or flag; blocked by modal → dismiss first; UI changed → update step. For standard controls, prefer converting to playwright-code with the correct label over rewording ai-action.

F3: Interaction Failure

Signals: element located but action had no effect; AI retried the same action. Classic offenders: native <select>, custom date/time pickers, drag-and-drop, rich text editors, small icon buttons, special key sequences. Fix: convert to playwright-code with verified selectors (selectOption(), picker interactions, fill(), keyboard APIs). This is the canonical playwright-first case.

F4: Assertion Failure

Signals: failing step is a verify/assert; expected content absent. Fixes: dynamic data → assert presence/format, not exact value; minor text drift → match actual text; below viewport → scroll first; timing → add gate before assertion; genuine app bug → report to the user, never "fix" the test to mask it.

F5: Navigation / State Error

Signals: wrong page in screenshot, unexpected URL/modal/login prompt. Fixes: handle the unexpected modal; check the login step completed; add a gate after the previous step; empty state → fixture data precondition.

F6: Timing Issue

Signals: intermittent pass/fail, "not yet visible", loading spinner in screenshot. Fix: add a verification gate between trigger and failing step — await expect(landmark).toBeVisible() in playwright-code, or ai-action "Verify [landmark] is visible". Prefer gates over arbitrary waits.

F7: Variable / Configuration Error

Signals: {{VARIABLE}} appears literally in events; value mismatched to field type. Fix: add/correct the variable definition (test case or project level); check name spelling between step text and variable. {{TIMESTAMP}} is not built-in — needs a RANDOM_STRING variable with a generation type.

F8: Playwright Code Error

Signals: "Timeout", "strict mode violation", "locator resolved to N elements" from a playwright-code step. Fixes: outdated selector → inspect actual page, update; multiple matches → { exact: true }, parent scoping (getByRole('row', { name: /…/ })), or .first(); not visible → scroll/gate first; wrong page → check the previous navigation. Do not fall back to ai-action for a deterministic interaction just because a selector broke — fix the selector.

F9: Environment / Infrastructure Error

Signals: DNS/network errors, "browser crashed", "target closed", ERR_CONNECTION_REFUSED. Do not modify the test case. Advise checking target app availability and runner status (list_runner_inventory), then re-run.

F10: Upstream-Cancelled Member — fix the cause, not this case

Signals: the run's status is CANCELLED (not FAIL) with a cancellationReasonCode of LOGIN_FLOW_FAILED, LOGIN_FLOW_CANCELLED, or EARLIER_CASE_FAILED; the error text says the test "did not run because its login flow … failed/was stopped" or "an earlier test case in the group failed". The member itself never executed. This case is not the defect. Do not edit it — its steps never ran. Call get_run_session(runSessionId) to find the member that actually failed (the login flow, or the earlier group case), then diagnose that member with F1–F8. Fixing the upstream failure unblocks every downstream member that was cancelled.

• USER_SINGLE / USER_GROUP cancellations are deliberate manual stops — nothing to fix.
• There is no SKIPPED status; a member that never ran is CANCELLED. Treat CANCELLED as "did not run", never as a pass.

Workflow

1. Gather Context

1. get_test_run with include: ["events", "artifacts"] (or parse the pasted report).
2. get_test_case for current steps, variables, targets, and last 5 runs — a previously passing test suggests UI change or environment; never-passed suggests a design flaw.
3. Find the first [ERROR] step; read all preceding events and the screenshots at and before the failure.

2. Classify

Decision order: CANCELLED by an upstream login-flow/earlier-case failure → F10 (diagnose the upstream member instead); infrastructure → F9; playwright-code error → F8; unresolved variable → F7; wrong page/modal in screenshot → F5; failing step is an assert → F4; "AI model" error → investigate per F1 before accepting it; locate attempts failed → F2; located but ineffective → F3; intermittent/"not visible" → F6. If ambiguous, say so and ask.

3. Investigate

• Check all {{VAR}} references against test-case and project variables first — the most common hidden cause.
• Check the screenshot from the step before the failure.
• If a browser tool is connected: reproduce the page state, inspect the element, and test the proposed selector live before proposing. Ask for a direct URL if the page needs deep navigation.
• No browser tool: ask for a screenshot of the failing screen with the element visible, or the selector from the user's DevTools.

4. Propose

## Diagnosis
**Failing step:** Step [N] — "[action text]"
**Classification:** [F1–F9]: [name]
**Root cause:** [concise]
**Evidence:** [events/screenshots supporting it]

## Proposed Fix
### Option A [recommended]
**Step [N]:** From `[current]` → To `[proposed]` (type: ai-action | playwright-code)
[Inserted steps, added variables, related steps to convert in the same update]
**Why this fixes it:** [root cause → fix]

Include the related-steps conversion batch (per Playwright-First Stabilization) when applicable.

5. Apply After Confirmation

1. update_test_case with corrected steps/variables/config (include activeRunResolution if runs are active).
2. Report exactly what changed.
3. Ask: "Re-run to verify the fix?"
4. Same step fails again → don't re-propose the same fix; investigate the new error. Different step fails → diagnose separately (expected when fixing multi-issue cases one at a time).

6. Recurring Failures

If a step keeps failing across attempts: check whether you're oscillating between approaches; get exact selectors from the user's DevTools or a live browser-tool session; if the flow is fundamentally un-automatable (CAPTCHA, email/OTP, file upload, multi-tab), flag it and recommend manual testing for that part.

Common Fix Patterns

Native dropdown → playwright-code

await page.getByRole('combobox', { name: 'Status' }).selectOption('Active');

Date picker → playwright-code

await page.getByRole('textbox', { name: 'Start Date' }).fill('2026-04-15');
// or, calendar popup:
await page.getByRole('textbox', { name: 'Start Date' }).click();
await page.getByRole('gridcell', { name: '15' }).click();

Icon button → scoped playwright-code (preferred over rewording)

await page.getByRole('row', { name: /School ABC/ }).getByRole('button', { name: 'Edit' }).click();

Missing scroll — insert ai-action "Scroll down to the bottom of the form" before the interaction.

Timing gap — insert a gate between action and next step:

await page.getByRole('button', { name: 'Save' }).click();
await expect(page.getByText('Record saved')).toBeVisible();

Dynamic data assertion — Verify the "Total Users" field shows a number instead of an exact value.

Unresolved variable — add { "name": "SEARCH_KEYWORD", "type": "VARIABLE", "value": "…" } to the test case (or project config if shared).

Strict mode violation — narrow with { exact: true }, parent scoping, or .first().

File upload — setInputFiles only works with attached test files; otherwise flag as un-automatable.

Batch Fix Mode

When asked to fix many failures: list_test_runs with status: "FAIL" → group by classification → let the user prioritize → fix one case at a time starting with the most common pattern → then offer: "The same fix likely applies to [X, Y] — apply it there too?"

What This Skill Does NOT Do

• Fix app bugs — report real failures; never adjust the test to mask them.
• Redesign test cases — if the approach is fundamentally wrong, recommend recreating via the skytest skill.
• Fix infrastructure — F9 failures are not test case issues.