expect

# Expect — Diff-Aware AI Browser Testing Analyze git changes, generate targeted test plans, and execute them via AI-driven browser automation. > **Note:** If `disableSkillShellExecution` is enabled (CC 2.1.91), the agent-browser install check won't run. Verify it's installed: `npx agent-browser --version`. ```bash expect # Auto-detect changes, test affected pages expect -m "test the checkout flow" # Specific instruction expect --flow login # Replay a saved test flow expect --target branch # Test all changes on current branch vs main expect -y # Skip plan review, run immediately ``` **Core principle:** Only test what changed. Git diff drives scope — no wasted cycles on unaffected pages. ## Argument Resolution ```python ARGS = "[-m <instruction>] [--target unstaged|branch|commit] [--flow <slug>] [-y]" # Parse from full argument string import re raw = "" # Full argument string from CC INSTRUCTION = None TARGET = "unstaged" # Default: test unstaged changes FLOW = None SKIP_REVIEW = False # Extract -m "instruction" m_match = re.search(r'-m\s+["\']([^"\']+)["\']|-m\s+(\S+)', raw) if m_match: INSTRUCTION = m_match.group(1) or m_match.group(2) # Extract --target t_match = re.search(r'--target\s+(unstaged|branch|commit)', raw) if t_match: TARGET = t_match.group(1) # Extract --flow f_match = re.search(r'--flow\s+(\S+)', raw) if f_match: FLOW = f_match.group(1) # Extract -y if '-y' in raw.split(): SKIP_REVIEW = True ``` ## STEP 0: MCP Probe + Prerequisite Check ```python # memory is alwaysLoad in .mcp.json (CC 2.1.121+, #1541) — probe below kept as fallback for older CC: ToolSearch(query="select:mcp__memory__search_nodes") # Verify agent-browser is available (Rust-native, no Playwright) Bash("command -v agent-browser || npx agent-browser --version") # If missing: "Install agent-browser: npm i -g agent-browser" # Load agent-browser's version-matched workflow guide (ships with the CLI). # NOT `skills get agent-browser` — that resolves but returns only the thin # top-level router; `core --full` is the actual 2,800+ line command reference. Bash("agent-browser skills get core --full") ``` ## CRITICAL: Task Management ```python # 1. Create main task IMMEDIATELY TaskCreate( subject="Expect: test changed code", description="Diff-aware browser testing pipeline", activeForm="Running diff-aware browser tests" ) # 2. Create subtasks for each pipeline phase TaskCreate(subject="Check fingerprint (skip if unchanged)", activeForm="Checking fingerprint") # id=2 TaskCreate(subject="Scan git diff and classify changes", activeForm="Scanning diff") # id=3 TaskCreate(subject="Map changes to routes/URLs", activeForm="Mapping routes") # id=4 TaskCreate(subject="Generate AI test plan", activeForm="Generating test plan") # id=5 TaskCreate(subject="Execute tests via agent-browser", activeForm="Executing browser tests") # id=6 TaskCreate(subject="Compile test report", activeForm="Compiling report") # id=7 # 3. Set dependencies for sequential phases TaskUpdate(taskId="3", addBlockedBy=["2"]) # Diff scan needs fingerprint check TaskUpdate(taskId="4", addBlockedBy=["3"]) # Route map needs diff results TaskUpdate(taskId="5", addBlockedBy=["4"]) # Test plan needs route map TaskUpdate(taskId="6", addBlockedBy=["5"]) # Execution needs test plan TaskUpdate(taskId="7", addBlockedBy=["6"]) # Report needs execution results # 4. Update status as you progress TaskUpdate(taskId="2", status="in_progress") # When starting TaskUpdate(taskId="2", status="completed") # When done — repeat for each subtask ``` ## Pipeline Overview ``` Git Diff → Route Map → Fingerprint Check → Test Plan → Execute → Report ``` | Phase | What | Output | Reference | |-------|------|--------|-----------| | **1. Fingerprint** | SHA-256 hash of changed files | Skip if unchanged since last run | `references/fingerprint.md` | | **2. Diff Scan** | Parse git diff, classify changes | ChangesFor data (files, components, routes) | `references/diff-scanner.md` | | **3. Route Map** | Map changed files to affected pages/URLs | Scoped page list | `references/route-map.md` | | **4. Test Plan** | Generate AI test plan from diff + route map | Markdown test plan with steps | `references/test-plan.md` | | **5. Execute** | Run test plan via agent-browser | Pass/fail per step, screenshots | `references/execution.md` | | **6. Report** | Aggregate results, artifacts, exit code | Structured report + artifacts | `references/report.md` | ## Phase 1: Fingerprint Check Check if the current changes have already been tested: ```python Read(".expect/fingerprints.json") # Previous run hashes # Compare SHA-256 of changed files against stored fingerprints # If match: "No changes since last test run. Use --force to re-run." # If no match or --force: continue to Phase 2 ``` Load: `Read("references/fingerprint.md")` ## Phase 2: Diff Scan Analyze git changes based on `--target`: ```python if TARGET == "unstaged": diff = Bash("git diff") files = Bash("git diff --name-only") elif TARGET == "branch": diff = Bash("git diff main...HEAD") files = Bash("git diff main...HEAD --name-only") elif TARGET == "commit": diff = Bash("git diff HEAD~1") files = Bash("git diff HEAD~1 --name-only") ``` Classify each changed file into 3 levels: 1. **Direct** — the file itself changed 2. **Imported** — a file that imports the changed file 3. **Routed** — the page/route that renders the changed component Load: `Read("references/diff-scanner.md")` ## Phase 3: Route Map Map changed files to testable URLs using `.expect/config.yaml`: ```yaml # .expect/config.yaml base_url: http://localhost:3000 route_map: "src/components/Header.tsx": ["/", "/about", "/pricing"] "src/app/auth/**": ["/login", "/signup", "/forgot-password"] "src/app/dashboard/**": ["/dashboard"] ``` If no route map exists, infer from Next.js App Router / Pages Router conventions. Load: `Read("references/route-map.md")` ## Phase 4: Test Plan Generation Build an AI test plan scoped to the diff, using the scope strategy for the current target: ```python scope_strategy = get_scope_strategy(TARGET) # See references/scope-strategy.md prompt = f""" {scope_strategy} Changes: {diff_summary} Affected pages: {affected_urls} Instruction: {INSTRUCTION or "Test that the changes work correctly"} Generate a test plan with: 1. Page-level checks (loads, no console errors, correct content) 2. Interaction tests (forms, buttons, navigation affected by the diff) 3. Visual regression (compare ARIA snapshots if saved) 4. Accessibility (axe-core scan on affected pages) """ ``` If `--flow` specified, load saved flow from `.expect/flows/{slug}.yaml` instead of generating. If NOT `--y`, present plan to user via `AskUserQuestion` for review before executing. Load: `Read("references/test-plan.md")` ## Phase 5: Execution ### agent-browser Quick Primer > Floor is `>= 0.31.1` (0.27.1 is documented broken on prod pages); current tested release is **0.38.1** (see `upstream-version-tested`). Commands below hold across this range. 0.30+ adds `agent-browser read` (agent-readable text extraction) and the `--restore` / `--namespace` session-restore workflow for stable, isolated browser state across agent runs. 0.33.0 adds `agent-browser a11y [url]`, an embedded axe-core audit (WCAG tag filtering, selector scoping, iframe-aware text/JSON output) available as both a CLI command and an MCP tool. 0.34.0 adds persistent session-to-tab binding for shared Chrome sessions, with `--pin-tab` making the binding strict so an externally closed tab yields a stable `tab_gone` error. 0.35.0 adds `--ca-cert <path>` for a trusted local CA behind an SSL-inspecting proxy (Linux-only; the targeted alternative to `--ignore-https-errors`). 0.35.1 changes behaviour the Diff row below relies on: `diff snapshot` now resets ref numbering per diff and invalidates refs across navigations, so refs captured before a navigation must be re-read rather than reused. 0.36.0 adds experimental WebMCP tool discovery (`webmcp list/invoke/result/cancel`, brief untrusted catalog summaries on `open`). 0.37.0 makes `record` capture the active page at 30 fps and inherits session setup into new tabs. 0.38.0 adds `snapshot --delta` (full baseline, then `unchanged` or a compact structural change), `screenshot --if-changed --threshold`, persistent refs that survive same-document DOM changes, `--human` / `--input-mode` pointer realism, and `auth login --no-navigate`. Full command surface: `references/upstream.md` and `agent-browser skills get core --full`. | Area | Command | Notes | |------|---------|-------| | Snapshot | `agent-browser snapshot -i` | ARIA tree w/ `@eN` refs. `-C`/`--cursor` was removed in 0.22 | | Snapshot delta *(0.38+)* | `agent-browser snapshot --delta` | Full tree once, then `unchanged` or a compact structural diff; refs persist across same-document changes | | Semantic locator | `agent-browser find role button click --name "Continue"` | Grammar: `find <locator> <value> [action]`; stable alternative to `@eN` refs | | Interaction | `fill @e1 "..."`, `click @e2`, `press Enter`, `drag @e1 @e2`, `upload @e1 file.pdf` | All take ARIA refs. Add `--human` for a curved approach instead of an instant jump | | Waits | `wait --load networkidle`, `wait --text "Success"`, `wait --fn "window.ready"` | Event-driven, never sleep-based | | Network | `network route "*analytics*" --abort`, `network route "https://api/*" --body '{...}'` | Intercept + stub | | State | `state save/load auth.json`, `--session-name <name>` | Persist auth across runs | | Vault | `vault store github_pat`, `vault load github_pat` | Encrypted credential store | | Diff | `diff snapshot`, `diff screenshot --baseline /tmp/x.png` | ARIA + pixel diffing across separate runs; prefer `snapshot --delta` for within-run before/after on the same page | | Capture | `screenshot --annotate`, `screenshot --if-changed --threshold <n>` *(0.38+)*, `pdf`, `record start/stop --fps <n>` | `--if-changed` skips re-encoding an unchanged screenshot; `record` defaults to 30 fps (0.37+) | | Dashboard | `agent-browser dashboard start` *(0.25+)* | Browser-side runtime inspector on :4848 | ### Run the test plan ```python expect_task = Agent( subagent_type="ork:expect-agent", prompt=f"""Execute this test plan: {test_plan} For each step: 1. Navigate to the URL 2. Execute the test action 3. Take a screenshot on failure 4. Report PASS/FAIL with evidence """, run_in_background=True, model="sonnet", max_turns=50 ) # Stream agent-browser progress line-by-line instead of polling (CC 2.1.98+) # Each stdout line from agent-browser arrives as a notification — useful for # catching a failing step early rather than waiting for the full plan. # Full pattern: Read("${CLAUDE_PLUGIN_ROOT}/skills/chain-patterns/references/monitor-patterns.md") Monitor(pid=expect_task.agent_id) # For long test plans (>3 min typical), notify on completion — requires # Remote Control + "Push when Claude decides" config (CC 2.1.110+). # Skip silently if the user doesn't have Remote Control enabled. if test_plan_duration_estimate > 180: PushNotification( message=f"ork:expect complete — {passed}/{total} steps passed on {len(affected_urls)} pages", status="proactive" ) ``` Load: `Read("references/execution.md")` ## Phase 6: Report ``` expect Report ═══════════════════════════════════════ Target: unstaged (3 files changed) Pages tested: 4 Duration: 45s Results: ✓ /login — form renders, submit works ✓ /signup — validation triggers on empty fields ✗ /dashboard — chart component crashes (TypeError) ✓ /settings — preferences save correctly 3 passed, 1 failed Artifacts: .expect/reports/2026-03-26T16-30-00.json .expect/screenshots/dashboard-error.png ``` Load: `Read("references/report.md")` ## Saved Flows Reusable test sequences stored in `.expect/flows/`: ```yaml # .expect/flows/login.yaml name: Login Flow steps: - navigate: /login - fill: { selector: "#email", value: "test@example.com" } - fill: { selector: "#password", value: "password123" } - click: button[type="submit"] - assert: { url: "/dashboard" }