verify

name	verify
description	End-to-end app testing pipeline for browser-based web apps on cloud browsers via Azure Playwright Workspaces: verify the live app, author tests for working flows, heal failing tests, then re-run with video and produce one consolidated report. Three phases (app verification, test authoring, test healing) plus a scoped final sanity run; each phase emits a sub-report. Use when asked to "verify my app", "make sure my app works and has good test coverage", "do a full QA pass", "add tests where I'm missing coverage", "fix failing tests and add new ones", "smoke test before deploy", "QA my deployment", or any combined verify-and-test ask. Requires Azure CLI (`az login`) and a Playwright Workspace resource ID — bootstraps configs on first invocation. Browser-based web apps with Playwright only — not for unit/API tests, native mobile/desktop, load/perf, or full accessibility audits.
license	MIT
allowed-tools	Bash(playwright-cli:*) Bash(npx:*) Bash(npm:*) Bash(git:*) Bash(az:*)

/verify — App Testing Pipeline

Drive a live app, classify what works and what doesn't, write tests for the working flows, fix any failing existing tests, re-run with video, and report. One skill, three phases plus a final sanity run, one consolidated report.

When to use

Invoke for any of these intents — Copilot CLI auto-discovers this skill on intent match. Three ways to trigger:

• Slash command: /verify (Copilot CLI exposes skills as slash commands by name).
• Natural language: "verify my app", "smoke test", "QA my deployment", "make sure my app works and has good test coverage", "verify, then add tests, then heal anything broken", "I just refactored — verify the app, add coverage for new flows, fix existing tests". Copilot routes based on the skill's description.

When NOT to use

Stop and say so for: native mobile / desktop apps (Swing, WinForms, iOS, Android), API-only services without a UI, load / performance benchmarking, full accessibility audits, debugging why source code misbehaves (this skill verifies whether flows work, not why they don't).

Hard dependencies

/verify runs cloud-only against an Azure Playwright Workspace. The user-side prerequisites (verify these before starting):

• Azure CLI installed + signed in:

  az account show
  

  If az is missing or az account show fails, halt with: "/verify needs the Azure CLI. Install it and run az login, then re-invoke /verify."

• A Playwright Workspace resource ID. The skill asks for this once at Step 0 if neither playwright.service.config.ts nor .playwright/cli.config.json exists yet — see ./references/cloud-setup.md.

Everything else (npm dependencies, the playwright-core override, the loopback patch, a minimal playwright.config.ts if the repo has no Playwright project at all, the cloud sibling configs, .gitignore) is handled by the skill automatically during Step 0. Do NOT ask the user for permission to install npm packages, modify package.json, create config files, or run npm init-style bootstrapping — those edits are part of the cloud-setup that the user implicitly opted into by invoking /verify. The only user input the skill ever asks for at Step 0 is the workspace resource ID (which cannot be inferred).

Safety-related asks during the pipeline still apply: don't run anything that looks destructive (data resets, wipes, drops, restores) without explicit approval; don't delete tests without explicit approval; surface suspected regressions to the user instead of fixing them silently. Those aren't setup prompts — they're runtime decisions where the user owns the call. Non-destructive setup (npm installs, frontend dev script, backend services like docker-compose up, non-destructive migrations) runs automatically — that's part of bringing up the app the user invoked /verify against.

Pipeline overview

Phase	What it does	Skipped when
1. App Verification	Drive the live app via playwright-cli, exercise the in-scope user flows (and the sub-flows they reveal), classify each step PASS / FAIL / BLOCKED. Surfaces broken flows BEFORE writing tests against them.	App URL can't be determined AND user declines to provide one.
2. Test Authoring	For flows that PASSed in Phase 1 (i.e. actually work), plan + author Playwright tests matching repo conventions, then iterate the new tests until green. Phase 2 owns its new tests end-to-end (plan + generate + heal-new).	No working uncovered flows.
3. Test Healing	Run the pre-existing test suite. For each failure, decide whether it's test drift (fix it) or a suspected app bug (stop and surface to the user — never mask app bugs). Does not touch tests Phase 2 just wrote.	No pre-existing tests in the repo.
Final Sanity	Scoped re-run with video on — Phase 2's new tests + Phase 3's healed tests + dependents of any shared files, OR all pre-existing tests when neither phase modified anything. The Playwright HTML report auto-opens in the user's browser with each video embedded inline.	The repo has no Playwright tests at all (no *.spec.ts files and Phase 2 didn't author any).

After all applicable phases run, emit a single consolidated report in chat — see ./references/consolidated-report.md.

Process

Step 0 — Repo discovery + cloud-config gate (always)

Before any phase work, build a deep mental model of the repo. The phases that follow are only as good as Step 0's understanding — shallow discovery = missed flows, missed regressions, weak tests. Follow ./references/repo-discovery.md:

• Source code — routes, components, page objects, business-logic hotspots (forms, mutations, auth, state machines). Don't just list — understand what user flows exist end-to-end, including flows hidden behind clicks (a "manage members" page only reachable after creating an org; a "review" form revealed after submitting a draft). Source code tells you these sub-surfaces exist; live exploration in Phase 1 confirms how to reach them.
• Existing tests — every *.spec.ts in testDir. Note what's covered vs not, the conventions, the shared helpers / page objects / fixtures, the auth setup.
• playwright.config.ts — baseURL, testDir, projects, fixtures, reporter, webServer.
• package.json — scripts (how the app is launched), devDependencies.
• CI configs (.github/workflows/*.yml, azure-pipelines.yml) — the canonical test command.

This deep read informs every later phase: Phase 1 plans which flows to drive; Phase 2 plans coverage against actual routes + uncovered behaviors; Phase 3 has context when reasoning about failures. Re-consult source as you work — don't rely solely on Step 0's one-shot read.

Starting the app — trust the dev server's stdout, not the configured baseURL. When Step 0 launches the frontend (and any backend services), start each long-running server async with a shellId, read its stdout for the URL it actually bound to, and update baseURL to match before any browser session opens. See ./references/app-launch.md. This matters because cloud browsers reach the local app via the loopback tunnel — wrong URL = silent timeout.

Then verify the cloud configs that every subsequent phase depends on:

• Configs missing (playwright.service.config.ts and/or .playwright/cli.config.json) → run the bootstrap flow in ./references/cloud-setup.md. This silently installs @playwright/cli, adds the playwright-core override, applies the loopback patch, asks the user once for the workspace resource ID, then writes both configs and updates .gitignore — all in one Step 0 batch.
• Configs present → refresh the accessKey token in .playwright/cli.config.json via az account get-access-token (one shell call). Re-apply the playwright-core loopback patch (idempotent — survives npm install re-runs that clobber it). Halt with a clear message if az fails — never proceed with a stale token.

Output a one-line discovery summary before proceeding (e.g. "Project uses Playwright 1.49 with tests under tests/e2e/, file naming <area>.<flow>.spec.ts, shared auth via tests/auth.setup.ts + storageState. Cloud configs detected, token refreshed (workspace: my-pw-ws, eastus).").

Phase 1 — App Verification

Goal: find what's broken in the app before writing tests, so authoring (Phase 2) only covers things that actually work.

Full playbook: ./references/phase-1-app-verification.md.

Emit a Phase 1 sub-report (format: ./references/verification-report.md) when this phase completes, BEFORE starting Phase 2. The user can intervene here if a real app issue needs their attention.

If all flows BLOCK or FAIL, stop the pipeline and surface to the user — Phase 2 / 3 can't proceed against a broken app.

Phase 2 — Test Authoring (for working uncovered flows)

Goal: add Playwright coverage for flows that work but aren't tested. Phase 2 owns its new tests end-to-end (plan + generate + heal-new): it only emits its sub-report when those tests are green (or surfaced for user decision).

Full playbook: ./references/phase-2-test-authoring.md.

Authoring rules in brief (see playbook for details):

• Match the repo's existing test conventions discovered in Step 0.
• Cover only flows Phase 1 marked PASS — never author against a flow that's currently broken.
• Aim for broad, deep coverage. Build a coverage inventory of distinct user-observable behaviors — primary flows, their sub-flows (dialogs, modals, forms revealed after clicks), validation paths, empty states, error states. Don't pre-filter aggressively; when in doubt, author the test.
• Use role-based locators (getByRole, getByLabel); test-IDs second-best; CSS / XPath last resort.
• Reference credentials only via process.env.<NAME> — see ./references/credential-handling.md.
• No hardcoded URLs in tests (use baseURL); no waitForTimeout / networkidle / page.evaluate as user-facing assertions.
• Run the new tests and iterate (fix authoring mistakes in place) until green. Stop iterating if you're not making progress, and surface the test with three choices (keep failing / test.fixme / delete with approval). Never silently delete.
• Phase 2 does not hand red tests to Phase 3.

Emit a Phase 2 sub-report (tests added, iteration outcomes, anything awaiting user decision) before starting Phase 3.

Phase 3 — Test Healing (for pre-existing failing tests)

Goal: fix pre-existing tests that are red, preserving each test's intent. Does not touch tests Phase 2 authored this session.

Full playbook: ./references/phase-3-test-healing.md.

Healing rules in brief:

• Run with --reporter=json (CLI flag — never modify the project's playwright.config.ts reporter setting).
• For headed runs (visible browser), use the appropriate sibling config — see ./references/project-config-conventions.md.
• For each failure, decide whether it's test drift (selector / text / URL changed in a way the app intends) → fix it; or a suspected app bug or environmental issue → stop and ask the user. Cross-reference Phase 1: a failing test for a Phase-1-FAIL flow is almost certainly a real app issue, not test drift.
• Never mask app bugs by relaxing assertions, adding skips, or deleting tests silently.
• Stop iterating once you're not making progress. Offer the user three choices: keep failing / quarantine with test.fixme / delete (only with explicit approval).

Emit a Phase 3 sub-report (heal summary) before the final sanity run.

Final Sanity Run (scoped re-run with video) — always runs when there are any tests

End-of-pipeline scoped re-run with video recording on. The Playwright HTML report auto-opens in the user's default browser — every test plays its video inline, with screenshots and per-step DOM snapshots.

Scope: every *.spec.ts Phase 2 created + every *.spec.ts Phase 3 healed + dependents of any shared files Phase 2/3 edited. When Phase 2/3 modified nothing but the repo has pre-existing tests, scope = the full pre-existing suite (the user wants to see green-state on video too).

Skip only when the repo has no Playwright tests at all (no *.spec.ts files and Phase 2 didn't author any).

Full playbook: ./references/final-sanity-run.md. The result feeds the Final Sanity subsection of the consolidated report — no separate sub-report.

Final — Consolidated report

After phases finish, output ONE markdown report (a) inline to chat and (b) saved to .verify/verify-report.md (gitignored — Step 0 adds this to .gitignore). The two are identical content; the inline copy is the immediate deliverable, the file is for share-in-chat / paste-into-PR / look-back-later.

Mid-pipeline sub-reports (Phase 1 / 2 / 3) are inline-only; the consolidated final does NOT re-embed them.

What the consolidated report covers:

• Banner: pipeline status, app URL, workspace + region, branch + commit SHA, duration, timestamp.
• Summary table: one row per phase, headline outcome only.
• What needs YOUR attention: priority-ordered action items.
• Tests touched this run: single table covering Phase 2 adds + Phase 3 heals, with a per-test video link column.
• HTML report URL (already auto-opened) + the "saved to .verify/" pointer.
• Suggested next steps.

Format spec: ./references/consolidated-report.md.

Cross-cutting rules (apply across all phases)

• Take the time each phase deserves. Quality over speed. Don't compress evidence per flow / per failure to "save turns" — each phase is only as good as the evidence it produces. Shallow Phase 1 → no coverage in Phase 2 → misclassified failures in Phase 3. For wide scopes (many flows, large suites, unfamiliar apps), lean on whatever progress-tracking Copilot CLI offers (todo lists, plan files, scratchpad) so multi-turn sessions stay organized and you don't lose what you've already covered. Better to deliver thorough work over more turns than rushed work in fewer.
• Cloud-only execution. All browser sessions and test runs go through the user's Azure Playwright Workspace. playwright-cli reads .playwright/cli.config.json (remote endpoint, refreshed each invocation). npx playwright test runs route through playwright.service.config.ts (or playwright.video.config.ts for Final Sanity, which extends the service config). See ./references/cloud-setup.md.
• Internal knowledge — upstream playwright-cli skill. The @playwright/cli npm package ships a skill folder at node_modules/@playwright/cli/skills/playwright-cli/ (a SKILL.md plus a references/ folder with browser-automation primitives, debug-attach, tracing, request mocking, storage state, generated-code workflow, and more). When a phase doc says "see the upstream playwright-cli skill", that's where to look — pick whichever sub-doc fits the task at hand. Treat the folder as internal reference material — not a separately-invokable skill. /verify is the only top-level skill this plugin exposes.
• Credentials are sacred. Reference only via process.env.<NAME>. Never echo / log / hardcode. Fail fast on missing env vars. TOTP via otpauth — see ./references/credential-handling.md.
• Sensitive data redaction. Beyond credentials: redact PII / customer IDs / internal hostnames / tokens before quoting evidence. See ./references/sensitive-data.md.
• Don't modify playwright.config.ts. Use sibling configs for video / headed overrides; CLI flags for reporter. The one exception: a single additive import 'dotenv/config'; if env vars are needed and missing. See .

Scope guards

Stop and say so when the user's ask is outside scope:

• Native mobile (iOS / Android) → not in scope.
• Desktop apps (Java Swing, WinForms, Electron native APIs) → not in scope.
• API-only services without a UI → use API-test tooling, not this.
• Cypress / TestCafe / Jest e2e / pytest e2e → this skill writes / heals Playwright only. Don't migrate frameworks unless explicitly asked.
• Load / performance / a11y audits → out of scope.

Out of scope (even for in-scope apps)

• Creating PRs — that's the caller's choice. Show git status --short if helpful.
• Editing app source code to fix regressions — surface regressions, don't fix the app yourself.
• Running the full test matrix when only specific failures need attention.
• Modifying CI config to make tests pass.

See also

• ./references/installation.md — what the user must provide vs what the skill bootstraps automatically
• ./references/repo-discovery.md — what to read before doing anything
• ./references/credential-handling.md — TEST_* env var rules, auth.setup.ts + storageState, TOTP
• ./references/cloud-setup.md — cloud-only setup, token refresh, sibling-config conventions
• ./references/project-config-conventions.md — never modify playwright.config.ts; sibling configs
• ./references/recovery.md — mid-run failure playbook
• ./references/sensitive-data.md — PII / internal URL / token redaction
• ./references/phase-1-app-verification.md — Phase 1 detailed playbook
• ./references/phase-2-test-authoring.md — Phase 2 detailed playbook
• ./references/phase-3-test-healing.md — Phase 3 detailed playbook
• ./references/final-sanity-run.md — scoped post-pipeline run with video recording
• ./references/verification-report.md — Phase 1 sub-report format
• ./references/consolidated-report.md — final report format
• Upstream playwright-cli skill — see the cross-cutting rule above for what lives in the upstream skill folder and where