browser-automation

name	browser-automation
description	Interactive browser automation and visual validation using Playwright MCP server and GitHub Copilot Agent Mode. Creates a bridge between the agent and a live browser session (including authenticated user context), with optional scripted Playwright test generation for repeatable checks. Triggers on "browser automation", "interactive browser", "playwright mcp", "test the ui", "ui test", "visual test", "visual regression", "check interface", "deployment smoke test", "post-deploy validation", "config validation".
user-invocable	true

Browser Automation — Interactive Bridge + Visual Validation Skill

You perform interactive browser automation using Playwright MCP tools in GitHub Copilot Agent Mode. This skill focuses on creating a live bridge between the agent and a running browser session so you can test authenticated flows with the current signed-in user, inspect UI behavior in context, and capture evidence. When needed, you then convert validated interactive flows into repeatable Playwright test scripts.

Prerequisites

1. Microsoft Edge Playwright MCP Server configured in VS Code:

   • The microsoft.playwright MCP server provides mcp_microsoft_pla_browser_* tools
   • Enable GitHub Copilot Agent Mode (Ctrl+Alt+I)
   • MCP tools will be available for live browser interaction
   • Tool names follow the pattern mcp_microsoft_pla_browser_<action> (e.g., mcp_microsoft_pla_browser_navigate, mcp_microsoft_pla_browser_click, mcp_microsoft_pla_browser_snapshot)

2. Interactive browser session available:

   • The Playwright MCP browser starts as a fresh session with no cookies or stored credentials
   • For protected apps (Dynamics 365, Power Apps, Azure Portal, etc.), you must run Phase 0 (Authentication) first
   • Navigate to the base domain to trigger the login redirect, let the user complete sign-in, then proceed
   • Once authenticated, the session persists for the browser lifetime — reuse it for all subsequent navigation

3. Playwright installed (optional, for repeatable scripts):

   npm init playwright@latest
   

4. Playwright configured for visual testing in playwright.config.ts (optional):

   use: {
     video: 'retain-on-failure',  // Record videos of failed tests
     screenshot: 'on',             // Screenshot on every action
     trace: 'retain-on-failure',   // Detailed traces for debugging
   }
   

Quick Start

Ask GitHub Copilot (Agent Mode):

"Use browser-automation on [your app URL] to walk through [specific flow]
in the live authenticated browser session, capture before/after screenshots,
log structured captions, and then generate a Playwright test from the validated steps."

Example:

"Use browser-automation for https://contoso.crm.dynamics.com and:
1. Creates a new Account record
2. Fills in required fields
3. Saves and verifies success
4. Captures screenshots and captions for review
5. Produces a repeatable Playwright spec from the interactive run"

Copilot will drive the live browser flow first, then produce automation artifacts for repeatable testing.

CRITICAL RULES

1. Authenticate first (Phase 0). For protected apps, navigate to the base domain first, let the user sign in, then navigate to the full target URL. Never type passwords or interact with MFA prompts.

2. Take a screenshot BEFORE and AFTER every action. This creates the visual evidence chain.

3. Visually inspect screenshots for layout correctness — don't just check functional outcomes. After taking a screenshot, actually LOOK at it. Check for:

   • Alignment issues — elements not lining up, inconsistent spacing, off-center content
   • Overflow/clipping — text or components cut off, hidden behind other elements, horizontal scrollbars appearing unexpectedly
   • Empty gaps — large blank areas where content should be, collapsed containers
   • Overlapping elements — headers over content, modals not properly layered, sticky elements covering interactive areas
   • Responsive breakage — columns not wrapping, cards squishing to unreadable widths, nav menus overflowing
   • Visual hierarchy — headings that look like body text, actions that don't look clickable, missing borders/separators
   • Data presentation — columns too narrow for their content, truncated text without tooltips, numbers/dates misaligned

   Do NOT treat screenshots as mere evidence artifacts to collect. They are your primary tool for catching visual bugs that functional tests miss. If something looks wrong in a screenshot, it IS wrong — investigate and report it.

   Accessibility snapshots (DOM snapshots) are NOT a substitute for screenshots. DOM snapshots show correct data and structure but completely miss layout issues:

   • Grid misalignment from null CSS Grid cells — DOM shows correct data but columns are shifted
   • Missing sort indicators — DOM shows columnheader but no button inside (no sort arrows)
   • Slots/cards not rendering in the right position — DOM shows all data loaded, grid appears empty
   • Overlapping elements, broken flex layouts, invisible overflow

   Always take a screenshot after any visual change and actually inspect it before claiming something works. A DOM snapshot saying "12 records loaded" means nothing if the screenshot shows an empty grid.

4. Log every action with the structured caption format. Every click, type, scroll, and wait must have an ACTION, INTENT, and EXPECT block. Read resources/caption-format.md.

5. Never skip the edge case checklist. After the happy path, run through edge cases. Read resources/edge-cases.md.

6. Protect authenticated context. Avoid unnecessary sign-out/navigation that invalidates user login state.

7. Review the FULL evidence — video/screenshots + captions, not just one artifact.

8. Persist workflow state to disk. Use the shared workflow-state skill (../workflow-state/SKILL.md) and keep state synchronized with manage_todo_list.

9. MSAL.js popup flows are incompatible with Playwright MCP. MSAL.js uses popup windows for authentication, but the Playwright MCP browser cannot interact with popups (it only sees the main page). If the app under test uses MSAL.js popup auth, you must either: (a) switch to MSAL redirect flow before testing, or (b) use a non-MSAL auth approach (e.g., Power Pages v1 implicit grant). Never attempt to automate MSAL popup login — it will hang indefinitely.

10. Clean up session state between test runs. If a prior test or auth flow left stale state (e.g., MSAL's interaction_in_progress flag in sessionStorage, leftover cookies, or cached tokens), clear it before starting a new flow. Use mcp_microsoft_pla_browser_evaluate to run sessionStorage.clear() or target specific keys. Stale interaction_in_progress flags are a common cause of "interaction already in progress" errors that block subsequent auth attempts.

11. Viewport testing is mandatory. Every browser testing session MUST include at least one test at a reduced viewport:

    mcp_microsoft_pla_browser_resize: { width: 800, height: 600 }
    

    Test for:

    • Content not collapsing (flexShrink issues)
    • Horizontal scroll where needed (DataGrid columns)
    • Pagination footer visible
    • Headers/toolbars not overlapping content

    Then reset to standard viewport:

    mcp_microsoft_pla_browser_resize: { width: 1280, height: 768 }
    

12. Keyboard navigation verification. For pages with scrollable content, verify keyboard navigation works:

    1. Click inside the scrollable container
    2. Press Tab to focus the scroll container
    3. Press arrow keys / Page Down to scroll
    4. Verify content scrolls as expected

    If the scroll container is not keyboard-focusable, add tabIndex={0} to it.

13. Power Platform admin UI verification. After setting up permissions, web roles, or site settings via API, verify the result in the admin UI using browser automation:

    1. Navigate to Power Pages admin center
    2. Check table permissions have correct web role associations
    3. Check site settings values match expectations
    4. Navigate to the actual site and test an API call

    This catches "silent failures" where the API returns 200 but the configuration is actually broken (e.g., associations cleared by deployment).

14. Final validation must include a full screenshot layout review. At the end of every browser testing session (whether milestone or final), take a full-page screenshot of every key screen and visually inspect each one for layout correctness. This is NOT optional — it is the final gate before declaring the test pass clean.

    For each screenshot, check and report:

    • Page fills the viewport correctly (no excessive whitespace, no unexpected scroll)
    • Grid/list headers are aligned with their columns
    • Pagination is visible without scrolling (sibling of scroll container, not inside it)
    • Forms have consistent field widths and label alignment
    • Command bars / action buttons are positioned correctly (above content, not floating)
    • Navigation (left nav or top nav) renders at the correct width/height
    • No content is hidden behind overlapping elements
    • At 800×600: layout adapts without horizontal overflow or content loss

    If ANY screenshot shows a layout issue, report it as a finding — do not silently pass it.

15. Always test locally before testing deployed. Browser automation follows a local-first approach. Test against the local dev server first, iterate until stable, deploy, then re-test against the deployed URL as a smoke test.

    Local testing catches most issues — layout, logic, data flow, navigation. Deployed smoke testing catches environment-specific issues — auth redirects, CORS, asset paths, permission configuration.

    Do NOT skip local testing and go straight to deployed testing. If a bug is found in the deployed environment, fix it locally, test locally, then redeploy.

Workflow

Before Phase 1, load ../workflow-state/SKILL.md, initialize workflow state, and update it at each phase transition and todo update.

Phase 0 — Authentication

Protected apps (Dynamics 365, Power Apps, Power Pages with auth, Azure Portal, etc.) require the user to sign in before the agent can interact with the application. The Playwright MCP browser is a fresh session with no stored cookies, so authentication must be handled as the very first step.

Authentication Flow

┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐
│ 1. Navigate to   │───▶│ 2. Detect login  │───▶│ 3. User completes│───▶│ 4. Navigate to   │
│    base domain   │    │    page & pick   │    │    password/MFA  │    │    target URL    │
│                  │    │    account        │    │                  │    │                  │
└──────────────────┘    └──────────────────┘    └──────────────────┘    └──────────────────┘
  e.g. org.crm.         Snapshot shows           Agent WAITS for        Once authenticated,
  dynamics.com          "Pick an account"        user confirmation      navigate to full URL

Step-by-Step

1. Navigate to the base domain (not the full app URL):

   mcp_microsoft_pla_browser_navigate → https://org.crm.dynamics.com
   

   Use only the base domain URL. The server will redirect to the identity provider login page (e.g., login.microsoftonline.com). Do NOT navigate to the full app URL yet — the login redirect can lose query parameters or fail with long URLs.

2. Take a snapshot to detect the login page state. The snapshot will show one of:

   • "Pick an account" — Multiple accounts are available. Identify the correct account from the list and click it using mcp_microsoft_pla_browser_click. Look for the account that matches the target org's tenant (e.g., [e-mail]).
   • "Enter password" — A single account was auto-selected. The user needs to enter their password manually.
   • "Sign in" — No cached accounts. The user needs to enter email + password.
   • Already authenticated — The app loaded directly (session/cookies were still valid). Skip to step 5.

3. If "Pick an account" is shown, click the correct account button:

   mcp_microsoft_pla_browser_click → ref for the matching account button
   

   After clicking, the page will transition to "Enter password" or directly authenticate (if SSO/Windows integrated auth is configured).

4. Ask the user to complete authentication. Tell the user:

   "The browser is showing the login page. Please enter your password (and complete MFA if prompted) in the browser window. Let me know once you're logged in."

   Then STOP and WAIT for the user to confirm they have logged in. Do NOT attempt to type passwords or interact with MFA prompts. Never ask the user for their password.

   After the user confirms, take a snapshot to verify:

   • The page URL has changed away from login.microsoftonline.com
   • The page title or content shows the authenticated app or app picker

5. Navigate to the full target URL:

   mcp_microsoft_pla_browser_navigate → https://org.crm.dynamics.com/main.aspx?appid={guid}&pagetype=entitylist&etn=...
   

   The authenticated session cookies are now set, so the full URL will load directly into the app without another login redirect.

6. Wait for the app to fully load. Dynamics 365 apps show a "Loading..." alert initially. Use mcp_microsoft_pla_browser_wait_for or take repeated snapshots until the main content (grid, form, dashboard) is visible. Then take an initial screenshot for evidence.

Authentication Tips

• Protect the session. Once authenticated, avoid navigating to external domains or clearing cookies. The session persists for the lifetime of the browser instance.
• If authentication fails (redirect loop, error page), take a screenshot, report the error to the user, and ask them to check their credentials or network.
• Multi-tenant scenarios. If the user has accounts in multiple tenants, the "Pick an account" page will list all of them. Match the account to the target org's domain.
• SSO (Windows Integrated Auth). In some enterprise environments, clicking the account may complete authentication automatically without a password prompt. Always take a snapshot after clicking to detect whether the user needs to act.
• Session expiry. Long-running test sessions may hit token expiry (typically 1 hour for Dynamics 365). If you encounter a surprise login redirect mid-test, repeat Phase 0.

Phase 1 — Plan the Test Run

Before generating code, define the test plan with the user:

Questions to ask:

1. What app? Get the URL (e.g., https://org.crm.dynamics.com/main.aspx?appid={guid})
2. What flows? List user journeys (e.g., "Create account → Add contact → Link opportunity")
3. What to check? Define expected visual states for each step
4. What deployment/config checks? Identify smoke checks (auth, routing, env-specific endpoints, static assets, key pages)
5. What edge cases? Which items from the checklist apply to this app?

Output: A test plan document with:

• Test scenarios (happy path + edge cases)
• Deployment/config smoke scenarios
• Expected behavior for each step (ACTION, INTENT, EXPECT_VISUAL)
• Screenshots to capture
• Assertions to make

Example Ask:

"Create a browser automation test plan for the Dynamics 365 Accounts app that tests:
- Creating a new account with required fields
- Saving and verifying the record appears in the view
- Editing the account name and phone
- Testing validation on empty required fields
Include edge cases for long text, special characters, and narrow browser width."

Phase 2 — Generate Playwright Test with Captions

Use GitHub Copilot Agent Mode with Playwright MCP to generate the test script:

Ask Copilot:

"Create a Playwright browser automation test for [app URL] that:
1. Tests the [specific user flow]
2. Takes screenshots before/after each action
3. Generates structured captions with ACTION, INTENT, EXPECT_VISUAL
4. Saves captions to JSON for evidence review"

Example Test Pattern (skeleton):

import { test, expect } from '@playwright/test';
import { writeFileSync } from 'fs';

test('browser automation: create account record', async ({ page }) => {
  const captions: Array<{ timestamp: string; event: string; action: string; intent: string; expectVisual: string }> = [];

  // Step 1: Navigate to app
  await page.screenshot({ path: 'screenshots/01-before-navigate.png' });
  await page.goto('https://org.crm.dynamics.com/main.aspx?appid={guid}');
  await page.waitForLoadState('networkidle');
  await page.screenshot({ path: 'screenshots/02-after-navigate.png' });
  captions.push({ timestamp: '00:00', event: 'NAVIGATE_TO_APP', action: 'Navigate to app', intent: 'Open Accounts', expectVisual: 'Accounts grid visible' });

  // Step 2: Click New button
  await page.screenshot({ path: 'screenshots/03-before-new-click.png' });
  await page.click('button[aria-label="New"]');
  await page.waitForSelector('input[aria-label="Account Name"]');
  await page.screenshot({ path: 'screenshots/04-after-new-click.png' });
  captions.push({ timestamp: '00:05', event: 'CLICK_NEW', action: 'Click New button', intent: 'Open new record form', expectVisual: 'Empty form with Account Name field' });

  // Step 3: Fill account name
  await page.screenshot({ path: 'screenshots/05-before-fill-name.png' });
  await page.fill('input[aria-label="Account Name"]', 'Contoso Corporation');
  await page.screenshot({ path: 'screenshots/06-after-fill-name.png' });
  captions.push({ timestamp: '00:08', event: 'FILL_NAME', action: 'Fill Account Name', intent: 'Enter required field', expectVisual: 'Account Name shows Contoso Corporation' });

  // Step 4: Save record
  await page.screenshot({ path: 'screenshots/07-before-save.png' });
  await page.click('button[aria-label="Save"]');
  await expect(page.locator('.formNotification')).toContainText('saved');
  await page.screenshot({ path: 'screenshots/08-after-save.png' });
  captions.push({ timestamp: '00:12', event: 'SAVE_RECORD', action: 'Click Save', intent: 'Persist record', expectVisual: 'Success notification, form in saved state' });

  writeFileSync('test-results/captions.json', JSON.stringify({ testName: 'create-account-record', captions }, null, 2));
});

Every step follows the same pattern: screenshot BEFORE → act → screenshot AFTER → push caption. This creates a complete visual evidence chain that can be reviewed without re-running the test.

See resources/caption-format.md for the full caption field specification.

Phase 3 — Edge Case Testing (Playwright)

After the happy path, generate additional Playwright tests for edge cases from resources/edge-cases.md. Follow the same screenshot + caption pattern from Phase 2. For each applicable edge case, log PASS or FAIL.

Phase 4 — Compile Evidence

After running tests, gather evidence. See resources/evidence-pipeline.md for the full pipeline.

Key artifacts: test-results/videos/*.webm, screenshots/*.png, test-results/captions.json, playwright-report/.

Review the collected artifacts (video, screenshots, captions, and Playwright report) to produce a consolidated findings report with severity, root cause, and recommended fixes.

For each step in the caption file:

1. STEP: Caption event name
2. STATUS: PASS | FAIL | PARTIAL | UNCLEAR
3. EXPECTED: What should have happened (from captions)
4. ACTUAL: What you observed in video/screenshots
5. ISSUE: If FAIL or PARTIAL, describe the discrepancy
6. SEVERITY: Critical | High | Medium | Low

Phase 5 — Report Results

Present findings using this structure:

## Browser Automation Report — [App Name]
**Date:** [date] | **Tests Run:** [count] | **Pass Rate:** [percentage]

### Test Summary
| Category | Total | Pass | Fail | Duration |
|---|---|---|---|---|
| Happy Path | [n] | [n] | [n] | [time] |
| Edge Cases | [n] | [n] | [n] | [time] |
| Visual Checks | [n] | [n] | [n] | [time] |

### Findings
For each finding: SEVERITY, Test, Step, Expected, Actual, Screenshot, Recommendation

**Example Finding:**

SEVERITY: High Test: create-account-record Step: SAVE_RECORD Expected: Success notification appears, form transitions to saved state Actual: No notification shown, Save button remains enabled Screenshot: screenshots/08-after-save.png Recommendation: Check form save handler for silent failures

### Top 5 Priority Fixes
1. [Most critical issue]

Running Playwright Tests

npx playwright test tests/browser-automation-create-account.spec.ts

References

• resources/caption-format.md — Structured caption specification
• resources/edge-cases.md — Edge case checklist (input, state, layout, navigation, performance, accessibility, security)
• resources/team-testing.md — Team parallel testing (runSubagent prompts, coordination, merge template)
• resources/evidence-pipeline.md — Playwright evidence pipeline and review integration
• ../workflow-state/SKILL.md — Shared on-disk state persistence

Edge Case Testing

After the happy path, systematically test edge cases. See resources/edge-cases.md for the full checklist covering input, state, layout, navigation, performance, accessibility, and security edge cases.

For each tested edge case, log: EDGE_CASE, STATUS (PASS/FAIL/PARTIAL/N/A), EXPECTED, ACTUAL, SCREENSHOT, SEVERITY, NOTE.

Team Testing Workflows

For large apps, use runSubagent to delegate to a team of testing agents for parallel coverage. See resources/team-testing.md for prompts, coordination workflows, and the merge template.

Role	Focus Area
Happy Path Tester	Core user flows — CRUD, navigation, key business processes
Edge Case Hunter	Empty states, bad input, boundary conditions, error handling
Visual Inspector	Layout, alignment, responsive behavior, accessibility

Each test step must:

1. Screenshot before action
2. Perform action using Playwright
3. Screenshot after action
4. Log caption with ACTION, INTENT, EXPECT_VISUAL

If any step fails or produces unexpected visual results:

• Capture the failure screenshot
• Log the discrepancy
• Include in your findings report

Visual Inspection Guide

For each screen, use Playwright to check:

// Take full page screenshot
await page.screenshot({ path: `inspect-${screenName}.png`, fullPage: true });

// Resize viewport and screenshot at different widths
await page.setViewportSize({ width: 1024, height: 768 });
await page.screenshot({ path: `${screenName}-1024px.png` });

// Test browser zoom
await page.evaluate(() => { document.body.style.zoom = '150%'; });
await page.screenshot({ path: `${screenName}-zoom150.png` });

Compile findings with specific screenshots showing each issue. Include pixel coordinates or CSS selectors for each problem area.