skytest

name

skytest

description

Create, update, run, and manage SkyTest test cases via MCP — directly from the user's instructions, with no separate explore/plan phase. Accepts short prompts ("add a test that creates a product and verifies it in the list") or exact step lists; fills in element detail from existing test cases, screenshots, or a quick browser-tool pass. Writes playwright-code steps for deterministic interactions (clicks, fills, dropdowns, datepickers) and ai-action for visual verification. Also handles section coverage design, project configs, runner inventory, run monitoring, and stop operations. For diagnosing a failed run, use skytest-fix.

SkyTest Skill

Turn user instructions into reliable SkyTest test cases and manage them via MCP. Work from whatever the user gives — a one-line request, exact steps, screenshots, or a section name. Fill gaps from project context first; ask the user only what you genuinely cannot infer.

Non-Negotiable Rules

• One test case per create_test_case / update_test_case call — never batch.
• Confirm before create, update, delete, or run. Bulk confirmation is fine for creation ("create all", "create 1–8, skip 9"); confirm deletes individually.
• Never hardcode credentials or URLs — vars['NAME'] in playwright-code, {{NAME}} in ai-action. Credentials live in project or test-case variables.
• Users give display IDs (e.g., AUTH-01-001); MCP tools need the internal id — resolve via list_test_cases first. After create, use the returned id.
• name format: [Section] Short description. testCaseId: follow the project's existing convention; default XXXX-YY-ZZZ (feature abbreviation + 2-digit section + 3-digit case, e.g., AUTH-01-001).
• Browser target names must be descriptive (e.g., "Admin Portal") — never "Primary Browser".
• No FILE variables via MCP. Never read, download, or process video files — ask for screenshots instead.
• Don't queue runs unprompted — the one exception: offer to validate the first case of a new batch with a run.
• stop_all_runs cancels QUEUED + PREPARING + RUNNING; stop_all_queues cancels QUEUED only.

Step Typing: Playwright-First

Validated in practice: deterministic interactions run faster and far more reliably as playwright-code than as ai-action. Type steps by this policy:

playwright-code — default for interactions:

• Clicking buttons, links, sidebar/menu navigation, tabs
• Filling text inputs and textareas, clearing fields
• Selecting from dropdowns (selectOption), comboboxes, custom date/time pickers
• Checkboxes, radios, toggles
• Login flows (always, when selectors are known)
• Keyboard sequences (Tab, Enter, Escape)
• Exact element/text assertions when the locator is known: await expect(...).toBeVisible()

ai-action — for what code can't express reliably:

• Visual or fuzzy verification ("Verify the order summary shows the product name and a price")
• Scrolling ("Scroll to the bottom of the page")
• Steps on screens whose structure is unknown and can't be captured
• Highly dynamic or canvas-like widgets where selectors are unstable

Selector Confidence Ladder

How sure you are about an element decides whether a step gets code:

1. Verified — you interacted with the element via a connected browser tool (Playwright MCP / Chrome DevTools MCP), or the user supplied the code, or an existing passing test case uses the same selector. Use playwright-code as-is.
2. Exact label known — visible label text from screenshots, an existing test case, or the user's description. For standard controls (buttons, labeled inputs, native selects), write playwright-code with role/label locators — getByRole('button', { name: 'Save' }) — and validate with a first run.
3. Unknown — no label, no screenshot, no browser access. Use ai-action. Never fabricate selectors.

playwright-code Sandbox Facts

• Globals: page, expect (timeout pre-configured), vars (resolved variables), setTimeout/setInterval. No imports, no Node APIs.
• Statements execute one at a time, each with its own timeout and trace line — a multi-statement step (e.g., a whole login or form fill) is fine and preferred over many one-line steps.
• Prefer getByRole / getByLabel / getByText; add { exact: true } when a name could partially match.
• Insert await expect(...).toBeVisible() between page transitions — it doubles as a wait gate.
• Enter the app via the target's configured url; navigate inside the app by clicking, not page.goto().

Step Writing Rules (both types)

• Atomic ai-action steps: never verify + act + verify in one ai-action step — split it. (playwright-code steps may group a logical action sequence.)
• Gate every transition: after each navigation or mutation, assert a landmark (heading, toast, updated value) before the next interaction.
• Scroll before below-fold interaction (ai-action scroll step); verify above-fold, scroll, verify below-fold.
• Assert consequences, not appearance: after create → record visible in list; after delete → record gone; after login → user identity visible; after error → message shown and input preserved.
• Dynamic values (timestamps, counts) → assert presence/format, not exact value. Static labels → exact text.
• Every {{VAR}} needs a matching variable (test-case or project level). {{TIMESTAMP}} is NOT built-in — declare a RANDOM_STRING variable.

MCP Operations

Tool	Purpose
list_projects / get_project	Projects and project-level configs
get_project_test_summary	Status breakdown (byStatus) and kind breakdown (byKind: TEST vs LOGIN_FLOW)
list_test_cases / get_test_case	Existing coverage; each item includes kind; full case with steps, configs, last 5 runs
create_test_case / update_test_case / delete_test_case	One case per call. create_test_case accepts kind (TEST default, or LOGIN_FLOW); update_test_case cannot change kind
run_test_case	Queue one run session (login-flow prefixes + the test); overrides: url, prompt, steps, browserConfig, requestedDeviceId, requestedRunnerId. Returned runId is the test member; watch the whole run via its runSessionId
run_test_group	Queue a GROUP run session: { projectId, testGroupId } → { sessionId }. 409 if the group is already running
list_test_runs / get_test_run	Monitor runs; each carries kind/runSessionId/sessionPosition/cancellationReasonCode. get_test_run also returns the rolled-up session. include: ["events", "artifacts"] for step detail
get_run_session / list_run_sessions	Read a run session's rolled-up status + per-member statuses; list a project's sessions (optionally by testGroupId)
manage_project_configs	Upsert/remove project-level configs in one call
list_runner_inventory	Runner/device inventory and Android selector options
stop_all_runs / stop_all_queues	Cancel runs (projectId required, optional reason)

Login Flows, Run Sessions & Test Groups

The execution model is one run = one member of a run session, not one standalone run. Three concepts:

• Login flow — a test case with kind = LOGIN_FLOW: a reusable login sequence. Author it with create_test_case (kind: "LOGIN_FLOW"). Other cases reuse it by setting loginFlowId on a browser target, so the login runs as a prefix and its post-login session is restored before the test — no need to repeat login steps in every case.
• Run session — run_test_case creates a SINGLE session: each referenced login flow runs first (kind = LOGIN_FLOW), then the test (kind = TEST). The returned runId is the test member.
• Test group — an ordered set of cases (with its own login sessions) run as one GROUP session via run_test_group.

"Done" means the session settled, not one member. A member reaching PASS does not mean the run finished — a later member may still be running or have failed. To judge completion, read the rolled-up status: get_test_run returns session.status, or call get_run_session(runSessionId) for every member's status. A member that never ran settles CANCELLED (with a cancellationReasonCode such as LOGIN_FLOW_FAILED or EARLIER_CASE_FAILED) — there is no SKIPPED. A CANCELLED member caused by an upstream failure is not a defect in that case; see skytest-fix.

A grouped case's kind is fixed while it belongs to a test group — update_test_case won't change kind, and the web UI rejects flipping it. Author the case with the right kind up front.

Workflow

1. Read the Request, Pick a Mode

• Exact-steps mode — the user described the flow, even briefly. The most common case. Expand their description into typed steps; don't ask them to re-specify what they already said.
• Coverage mode — the user named a section without steps ("cover the Products section"). Apply the decomposition pattern below and present the case list for one confirmation.
• Management mode — update/run/delete/configs/stop on existing cases. Skip to step 6.

2. Inherit Project Conventions — Ask Only What You Can't Infer

1. list_projects → confirm the target project (skip if obvious from context).
2. get_project → project-level configs. Reuse BASE_URL, LOGIN_EMAIL, LOGIN_PASSWORD, etc. — never duplicate them as test-case variables. Tell the user what's being reused.
3. list_test_cases → existing coverage. If a similar case exists, propose updating it instead of duplicating.
4. If the project already has test cases, get_test_case on a representative one and inherit: browser target name, viewport, starting URL, the login step (copy verbatim — type and all), ID convention, variable naming. Do not re-ask the user for things existing cases already establish.
5. Only for a project's first test cases, confirm in ONE batched question: target name, viewport (suggest by app type: 1920×1080 desktop admin, 390×844 mobile), starting URL, credential variables, ID convention.
6. Android: list_runner_inventory; confirm { mode: "connected-device", serial } or { mode: "emulator-profile", emulatorProfileName }. If the server warns a device wasn't found, stop and confirm.

3. Fill In Element Detail — Only Where Needed

For steps at confidence level "unknown":

• Browser tool connected (Playwright MCP / Chrome DevTools MCP): walk only the screens the flow touches, capture exact roles/names from snapshots, and verify tricky interactions live before writing them into steps. Capture the login flow once and reuse it across every case.
• No browser tool: ask the user for screenshots of just the unclear screens, or fall back to ai-action for those steps.

Never explore the whole app — only the screens in the flow under test.

4. Check Automation Boundaries, Then Draft

SkyTest automates only what happens inside a clean browser session or installed Android APK. Cannot be automated: email/OTP/SMS, CAPTCHA, third-party auth or payment popups leaving the app domain, file uploads, download verification, multi-tab/new-window flows, push notifications, backend-only checks, external hardware. Split the flow around these or flag for manual testing — never create a case that fails by design. Iframes: proceed with caution and flag.

Present each draft compactly: name, ID, target, numbered steps with type per step, new variables, reused project variables. For batches, present all and take bulk confirmation.

5. Create and Validate

• create_test_case, one per call, only after confirmation.
• Pre-flight per call: single testCase object · name/ID formats · descriptive target name · consistent target IDs across steps · every {{VAR}} has a variable · every RANDOM_STRING has a value generation type · no hardcoded credentials · correct step type · no FILE variables · no duplication of project-level configs.
• After the first case of a new batch, offer one validation run — it catches systemic issues (login code, base URL, missing variables) before they're duplicated across the batch. Fix before creating the rest.

6. Manage

• Update: update_test_case (fields: name, url, prompt, steps, browserConfig, configs/variables, removeConfigNames/removeVariableNames). If active runs exist, include activeRunResolution: cancel_and_save or do_not_save.
• Environment-wide changes (base URL, credentials): manage_project_configs, not per-case edits.
• Delete: irreversible — removes the case plus all runs, files, configs. Confirm individually.
• MCP errors: report the full message; don't auto-retry. Common: project not found, duplicate display ID, active runs blocking update, invalid payload.

Coverage Mode: Section Decomposition

For CRUD admin / CMS sections (list view + detail + create/edit/delete), decompose one concern per test case:

#	Case	Focus
1	Navigation smoke (one per app)	Visit every main page via nav; assert heading + landmarks
2	List & detail	Assert list layout, search a fixture record, open detail, assert fields
3	Pagination	Page 2 shows different records; back to page 1 restores (skip if none)
4	Columns & sorting	Headers present; sort toggles order (skip if not sortable)
5	Create	Fill form with variables, submit, assert record in list; delete to clean up
6	Create → Edit	Create as setup, edit fields, assert updated values; clean up
7	Create → Delete	Create as setup, delete with confirm dialog, assert gone from list

Principles: mutation tests focus on exactly one operation — creation inside edit/delete tests is setup, not subject. Variable-first: every dynamic value is a {{VARIABLE}}; unique fields use RANDOM_STRING with TIMESTAMP_DATETIME. Read tests (2–4) assert structure against fixture data; mutation tests (5–7) assert the entered variable values persisted. Self-cleanup when the UI allows deletion. Read-only sections: cases 1–4 only. Adapt for filters, bulk actions, wizards — same principles, one concern per case.

Other section shapes (storefront, wizard/onboarding, dashboard, settings) follow the same principles without a fixed template.

Config Rules

• Names UPPER_SNAKE_CASE. VARIABLE for credentials (masked: true for passwords), URL for base URLs, APP_ID for Android app IDs.
• RANDOM_STRING must include a value generation type: TIMESTAMP_DATETIME (YYYYMMDDHHmmssSSS), TIMESTAMP_UNIX, or UUID.
• group field: include when the project uses team-scoped config groups.
• Test-case variables only for values not already in project config.

Target Defaults

• Browser: { url: "...", width: 1920, height: 1080 }
• Android: { type: "android", deviceSelector: {...}, appId: "...", clearAppState: true, allowAllPermissions: true }

Payload Example (playwright-first)

{
  "projectId": "proj_123",
  "testCase": {
    "name": "[Products] Create Product",
    "testCaseId": "PROD-01-001",
    "browserTargets": [
      { "id": "browser_a", "name": "Admin Portal", "url": "https://admin.myapp.com/login", "width": 1920, "height": 1080 }
    ],
    "steps": [
      {
        "id": "step_1",
        "target": "browser_a",
        "action": "await expect(page.getByRole('heading', { name: 'Sign In' })).toBeVisible();\nawait page.getByRole('textbox', { name: 'Email address' }).fill(vars['LOGIN_EMAIL']);\nawait page.getByRole('textbox', { name: 'Password' }).fill(vars['LOGIN_PASSWORD']);\nawait page.getByRole('button', { name: 'Sign In' }).click();\nawait expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();",
        "type": "playwright-code"
      },
      {
        "id": "step_2",
        "target": "browser_a",
        "action": "await page.getByRole('link', { name: 'Products' }).click();\nawait expect(page.getByRole('heading', { name: 'Products' })).toBeVisible();\nawait page.getByRole('button', { name: 'New Product' }).click();",
        "type": "playwright-code"
      },
      {
        "id": "step_3",
        "target": "browser_a",
        "action": "await page.getByRole('textbox', { name: 'Product name' }).fill(vars['PRODUCT_NAME']);\nawait page.getByRole('combobox', { name: 'Category' }).selectOption('Accessories');\nawait page.getByRole('button', { name: 'Save' }).click();",
        "type": "playwright-code"
      },
      {
        "id": "step_4",
        "target": "browser_a",
        "action": "Verify a success message appears and the product '{{PRODUCT_NAME}}' is visible in the products list",
        "type": "ai-action"
      }
    ],
    "variables": [
      { "name": "PRODUCT_NAME", "type": "RANDOM_STRING", "value": "TIMESTAMP_DATETIME" }
    ]
  }
}

LOGIN_EMAIL / LOGIN_PASSWORD are project-level configs — not duplicated in variables.

Final Report

After completing operations, summarize: created cases (with IDs), reused project variables, updated cases and fields changed, skipped cases and reasons (including manual-testing flags), stop/delete actions, and recommended next coverage.