菜单
WHEN: qa-semantic-csv-orchestrate or run_semantic_csv_eval dispatches an automation step that requires web UI interaction or assertion. Chrome DevTools Protocol: launch(), navigate(), interact(click/type/scroll), screenshot(), getDOM(), teardown().
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
| name | eval-driver-web-cdp |
| description | WHEN: qa-semantic-csv-orchestrate or run_semantic_csv_eval dispatches an automation step that requires web UI interaction or assertion. Chrome DevTools Protocol: launch(), navigate(), interact(click/type/scroll), screenshot(), getDOM(), teardown(). |
| type | rigid |
| requires | ["brain-read"] |
| version | 1.0.2 |
| preamble-tier | 3 |
| triggers | ["eval web UI","run browser eval","CDP eval driver","web UI eval"] |
| allowed-tools | ["Bash","AskUserQuestion","mcp__*"] |
Runner dispatch: qa-semantic-csv-orchestrate / run_semantic_csv_eval.py routes Surface: web rows in qa/semantic-automation.csv to this driver. Do not invoke this skill directly unless you are implementing or debugging the runner.
Automates browser interactions and state inspection using Chrome DevTools Protocol. Provides a programmatic interface for launching headless Chrome, navigating URLs, interacting with UI elements, capturing screenshots, and extracting DOM state.
This skill lists AskUserQuestion in allowed-tools — canonical for Claude Code and skill lint. Map to the host’s blocking interactive prompt per skills/using-forge/SKILL.md Blocking interactive prompts (Cursor AskQuestion; hosts without the tool: numbered options + stop). See using-forge Interactive human input (e.g. when choosing Appium MCP vs local CDP per CLAUDE.md D5).
| Rationalization | Why It Fails |
|---|---|
| "If the page loads, the eval passes" | A page can load with empty data, error state, or partial render. Load event fires before content is populated. Every scenario must assert on specific content. |
| "CSS selectors are fine for targeting elements" | CSS selectors break on visual refactors that don't change behavior. Use data-testid, ARIA roles, or labels. Test IDs are contracts; class names are not. |
| "teardown() can be skipped if the test fails" | An unclosed Chrome process holds the debug port. The next scenario cannot connect. teardown() must run in all paths — success, failure, and timeout. |
| "Screenshots are optional evidence" | If an assertion fails and there is no screenshot, debugging the failure requires re-running the scenario. Capture evidence every time. |
| "Timing issues are flakiness, not bugs" | Timing issues are bugs in the eval script. Use explicit wait conditions (networkIdle, element visible) — not fixed sleeps — so failures are deterministic. |
EVERY CDP SCENARIO FOLLOWS: launch() → navigate() → wait-for-load → interact → assert-specific-content → screenshot → teardown(). teardown() IS CALLED IN ALL PATHS. NO ASSERTION IS NON-SPECIFIC. NO INTERACTION HAPPENS BEFORE LOAD STATE IS CONFIRMED.
If you notice any of these, STOP and do not proceed:
navigate() is called without waiting for networkIdle or DOMContentLoaded — Interacting with a page that has not finished loading produces false "element not found" failures. STOP. Always verify load state after navigate() before any interaction.document.querySelector instead of accessibility identifiers or test IDs — CSS selectors break on UI refactors that don't change behavior. STOP. Elements must be targeted by data-testid, accessibility role, or stable aria-label.teardown() is not called after the scenario completes — An unclosed Chrome process holds a debug port that prevents the next scenario from launching. STOP. teardown() must be called in all paths — success, failure, and timeout.screenshot() call must produce a file path entry in the scenario output.getDOM() returning non-empty rather than specific content — A non-empty DOM matches any rendered page, including error pages. STOP. Every assertion must verify specific text, element state, or attribute value — not merely presence.launch() time to match the spec's target device class.launch() / CDP connection attempted with no browser process and no --remote-debugging-port listener — STOP. Complete Preflight below first; BLOCK with qa/logs/eval-preflight-*.log attachment.Before launch() or any CDP WebSocket connect:
mkdir -p ~/forge/brain/prds/<task-id>/qa/logs (see skills/forge-brain-layout/SKILL.md qa/logs/).which / common paths: google-chrome-stable, google-chrome, chromium, chromium-browser, microsoft-edge (distro-dependent); on macOS, /Applications/Google Chrome.app/.... Append --- web --- section + command output to eval-preflight-<ISO8601>.log.AskUserQuestion / AskQuestion: which browser binary + profile (headless vs headed) + --remote-debugging-port=<port> (must match driver config). Do not assume Chrome if only Chromium exists.--remote-debugging-port=... (and --user-data-dir if isolated profile needed); verify port listens (ss, lsof, or HTTP to /json/version) before scenarios — log failures.node -v, npm -v). Install MCP per upstream docs; run npx playwright install (or project-local install) for browsers. On missing Node, tell the user to install Node LTS, then retry — log stderr to qa/logs/.MUST elicit with a blocking interactive prompt per using-forge how web UI eval should run before treating any stack as decided — AskQuestion / numbered 1–3 + stop, not prose-only which stack?:
chrome-remote-interface / minimal driver (matches the API shape in this skill). Requires a running browser with --remote-debugging-port (Preflight above).npx playwright install for bundled browsers.eval-judge needs.If both MCP and a local CDP path exist, do not assume — same blocking interactive fork (MCP vs local CDP) and record the choice (e.g. in brain/prds/<task-id>/ notes) so runs are reproducible.
This skill enables eval scripts to drive web UI automation through CDP, supporting:
When a step lists DependsOn: <stepId>, the driver must resolve the prior step's result before executing:
semantic-eval-run.log — find the JSON line where stepId matches the dependency.FAIL or BLOCKED_DEPENDENCY: mark this step as BLOCKED_DEPENDENCY, write to run.log, skip CDP execution, continue to next step.PASS but result is {} (empty): mark this step as CONTEXT_GAP, log a warning ("dependency passed but returned no result data"), and attempt execution without interpolation.PASS and result has data: resolve any ${stepId.result.fieldName} references in the step's Intent or selector fields before running CDP commands.{ "userId": "abc123" }, replace ${prior-step.result.userId} with abc123 in this step's CDP payload before executing.semantic-eval-run.log as a JSON line with the reason before moving to the next step.Signature:
async launch() → Promise<Browser>
Description: Launches Chrome in headless mode and establishes CDP connection. Initializes browser instance for subsequent operations.
Behavior:
--headless flag--no-sandbox1920x1080Options:
{
headless: true,
args: [
'--no-sandbox',
'--disable-setuid-sandbox',
'--disable-gpu',
'--disable-dev-shm-usage',
'--window-size=1920,1080'
],
defaultViewport: {
width: 1920,
height: 1080
}
}
Error Handling:
Returns:
{
// CDP protocol interface
// Supports: Page, DOM, Input, Runtime, Network, etc. domains
}
Example:
const browser = await launch()
// Browser ready for navigation
Signature:
async navigate(browser: Browser, url: string) → Promise<{
loaded: boolean,
title: string,
url: string,
statusCode: number
}>
Description: Navigates to a URL and waits for page load completion. Blocks until DOM ready or timeout.
Behavior:
Load Conditions:
Error Handling:
{loaded: false} if page load timeoutReturns:
{
loaded: true|false, // True if all load conditions met
title: string, // Document title
url: string, // Final URL (post-redirects)
statusCode: number // HTTP status (200, 404, etc.)
}
Example:
const nav = await navigate(browser, 'http://localhost:3001/login')
if (!nav.loaded) {
throw new Error(`Page failed to load: ${nav.statusCode}`)
}
console.log(`Loaded: ${nav.title}`)
Signature:
async interact(
browser: Browser,
action: 'click'|'type'|'scroll',
selector: string,
value?: string
) → Promise<{
success: boolean,
message: string,
elementFound: boolean,
elementVisible: boolean
}>
Description: Performs user interactions: click elements, type text, scroll page. Actions wait for element visibility before execution.
Actions:
value (pixels)Element Selection:
input[name=email], .btn-submit, #modal > button)Visibility Check:
display: none or visibility: hiddenError Handling:
{success: false, elementFound: false} if selector doesn't match{success: false, elementVisible: false} if not visible after timeoutReturns:
{
success: boolean, // Action completed without error
message: string, // Descriptive status (e.g., "Clicked successfully")
elementFound: boolean, // Selector matched at least one element
elementVisible: boolean // Element was visible when action attempted
}
Examples:
// Click submit button
await interact(browser, 'click', 'button[type=submit]')
// Type email
await interact(browser, 'type', 'input[name=email]', 'user@example.com')
// Scroll down 500px
await interact(browser, 'scroll', null, '500')
Signature:
async screenshot(
browser: Browser,
filename: string
) → Promise<{
path: string,
size: number,
format: 'png'|'jpeg',
width: number,
height: number
}>
Description: Captures current page state as image file. Supports PNG (default) and JPEG formats.
Behavior:
screenshots/ directory (auto-created)Format Detection:
.jpg, .jpeg → JPEG (quality: 80%).png → PNG (lossless)Path Resolution:
screenshots/Error Handling:
screenshots/ directory if missingReturns:
{
path: string, // Absolute path to saved file
size: number, // File size in bytes
format: 'png'|'jpeg', // Image format used
width: number, // Screenshot width in pixels
height: number // Screenshot height in pixels
}
Example:
const shot = await screenshot(browser, 'login-form.png')
console.log(`Screenshot saved: ${shot.path} (${shot.size} bytes)`)
Signature:
async getDOM(
browser: Browser,
selector: string
) → Promise<{
elements: Array<{
tag: string,
id: string,
classList: string[],
text: string,
html: string,
attributes: Record<string, string>,
visible: boolean,
rect: {x: number, y: number, width: number, height: number}
}>,
count: number,
error: string|null
}>
Description: Extracts DOM state for matching elements. Returns element properties, text content, attributes, and visibility. Useful for assertions and state validation.
Behavior:
Selector Support:
div.modal, input[type=password], a:nth-child(2)elements[0] for first element:not(), :first-child, etc. (CSS4 support)Visibility Logic:
offsetHeight > 0 && offsetWidth > 0 (rendered)getComputedStyle(el).visibility !== 'hidden' (not hidden)getComputedStyle(el).display !== 'none' (not display:none)Error Handling:
{elements: [], error: '...'} if selector invalid{elements: []} if selector matches nothingReturns:
{
elements: [
{
tag: 'input',
id: 'email-field',
classList: ['form-control', 'required'],
text: '',
html: '<input type="email" name="email" ...>',
attributes: {
type: 'email',
name: 'email',
placeholder: 'Enter email'
},
visible: true,
rect: {x: 100, y: 200, width: 300, height: 40}
}
],
count: 1,
error: null
}
Examples:
// Get success message
const result = await getDOM(browser, '.success-message')
if (result.elements.length > 0) {
console.log(result.elements[0].text)
}
// Check if error is visible
const errors = await getDOM(browser, '.error')
assert.equal(errors.count, 0, 'Should have no errors')
Signature:
async teardown(browser: Browser) → Promise<void>
Description: Closes browser and releases resources. Ensures graceful shutdown and cleanup.
Behavior:
Error Handling:
Example:
try {
// ... test operations ...
} finally {
await teardown(browser)
}
Uses Chrome DevTools Protocol v1.3+ for automation:
Domains Used:
Page - Navigation, load events, viewportRuntime - JavaScript execution, DOM queriesInput - Mouse, keyboard eventsNetwork - Request/response monitoring, idle detectionDOM - Element queries (if needed for advanced scenarios)Key Events:
Page.loadEventFired - Window load completePage.domContentLoaded - DOM readyNetwork.requestWillBeSent / Network.responseReceived - Network activityRuntime.executionContextCreated - JS context readyUses WebSocket connection to local Chrome debugging port (default: 9222):
{
wsEndpoint: 'ws://localhost:9222/devtools/browser/...'
}
| Operation | Timeout |
|---|---|
| Browser launch | 30s |
| Page navigation | 10s |
| Element find | 5s |
| Element click | 5s |
| Element type | 5s |
| Network idle | 2s (inactivity) |
| Browser close | 5s |
DANGER: These beliefs cause flaky, unmaintainable eval scripts.
WRONG: Flakiness is a signal of bad test design, not a reason to avoid UI tests.
WRONG: Visual screenshots alone cannot verify logic, state, or assertions.
getDOM() to verify text, attributes, and element count before taking screenshotsWRONG: Page load completion does not guarantee element visibility or interactivity.
wait_for: 'selector_visible') and clickability checks before interactionWRONG: Clicks fail silently when element is not clickable (off-screen, animation in progress, handler not attached).
success and elementVisible return values. Retry with backoff on failure. Verify element is clickable (not just visible) before clickingWeb UI automation encounters these systematic challenges. Understanding them prevents flaky tests.
Problem: Element exists in DOM but is not yet visible or clickable.
display: none or visibility: hidden)overflow: hiddengetDOM() returns count > 0 but visible: falseinteract() returns {success: false, elementFound: true, elementVisible: false}wait_for: 'selector_visible' before interactdisplay, visibility, opacity, getBoundingClientRect()interact(browser, 'scroll', selector) or JavaScript scrollExample Fix:
// WRONG: Assumes element is ready
await interact(browser, 'click', '.submit-btn')
// RIGHT: Wait for visibility + check result
const waitResult = await getDOM(browser, '.submit-btn')
if (!waitResult.elements[0]?.visible) {
// Wait 500ms for CSS to apply
await new Promise(r => setTimeout(r, 500))
// Try again with scroll
await interact(browser, 'scroll', '.submit-btn')
}
const clickResult = await interact(browser, 'click', '.submit-btn')
if (!clickResult.success) {
throw new Error(`Click failed: visible=${clickResult.elementVisible}`)
}
Problem: Page loads (network idle) but JavaScript framework is still hydrating or lazy-loading content.
loaded: true, but interactive elements don't exist yet. Test flakes because element appears ~500ms laterdata-testid attributes that appear AFTER hydrationExample Fix:
// WRONG: Assumes load event means everything is ready
const nav = await navigate(browser, url)
const form = await getDOM(browser, 'form[data-testid=login]')
if (form.count === 0) {
throw new Error('Form not found') // Flaky - sometimes it appears after 500ms
}
// RIGHT: Retry with backoff until element appears
async function waitForElement(browser, selector, maxWait = 5000) {
const startTime = Date.now()
let attempts = 0
while (Date.now() - startTime < maxWait) {
const result = await getDOM(browser, selector)
if (result.count > 0 && result.elements[0].visible) {
return result.elements[0]
}
attempts++
const backoffMs = Math.min(100 * Math.pow(2, attempts), 1000)
await new Promise(r => setTimeout(r, backoffMs))
}
throw new Error(`Element ${selector} not found after ${maxWait}ms`)
}
const nav = await navigate(browser, url)
const form = await waitForElement(browser, 'form[data-testid=login]')
Problem: Element was found and cached, but removed from DOM before interaction.
getDOM() returned element at time T, but by time T+100ms, parent was unmounted and element is gone. Click fails with "elementFound: false"{isOpen && <Modal/>} became false)getDOM() succeeds, but next interact() fails with elementFound: falseExample Fix:
// WRONG: Cache DOM result, then click later
const buttons = await getDOM(browser, '.save-btn')
if (buttons.count === 0) throw new Error('Save button not found')
await new Promise(r => setTimeout(r, 200)) // Some async operation
await interact(browser, 'click', '.save-btn') // Might fail - parent unmounted
// RIGHT: Query just before interact, verify stability
async function clickStable(browser, selector) {
// Verify element exists and is visible
const result = await getDOM(browser, selector)
if (result.count === 0 || !result.elements[0].visible) {
throw new Error(`Element ${selector} not ready`)
}
// Click immediately after verification
const clickResult = await interact(browser, 'click', selector)
if (!clickResult.success) {
// Element was removed or hidden between check and click - retry
return await clickStable(browser, selector)
}
return clickResult
}
await clickStable(browser, '.save-btn')
Problem: Element exists and is visible, but is off-screen (below fold) so click fails or scrolls to wrong position.
overflow: hidden or overflow: auto affecting scroll contextgetDOM() shows visible: true and rect shows negative Y, or Y > viewport heightinteract() but verify)getDOM() rect to verify element is within viewport boundsExample Fix:
// WRONG: Trust that interact() handles scroll
await interact(browser, 'click', '.bottom-button') // Might click wrong element
// RIGHT: Verify scroll position before clicking
async function clickWithScroll(browser, selector) {
const result = await getDOM(browser, selector)
if (result.count === 0) throw new Error(`Element ${selector} not found`)
const rect = result.elements[0].rect
if (rect.y < 0 || rect.y + rect.height > 1080) {
// Element off-screen, scroll to center
await interact(browser, 'scroll', null, Math.round(rect.y - 540).toString())
// Wait for scroll settle
await new Promise(r => setTimeout(r, 200))
}
return await interact(browser, 'click', selector)
}
await clickWithScroll(browser, '.bottom-button')
Problem: Click action fires, but event handler has not been attached yet; click is processed but does nothing.
success: true), element doesn't change state. Click listener wasn't attached yet due to async script loadinginteract() returns {success: true} but page state unchangedwindow.React or __NUXT__)Example Fix:
// WRONG: Click and hope handler is attached
await interact(browser, 'click', '.submit')
const result = await getDOM(browser, '.success-msg')
if (result.count === 0) {
// Did handler fire? Or was it already gone?
throw new Error('No success message')
}
// RIGHT: Verify action had effect, retry if not
async function clickWithEffect(browser, clickSelector, resultSelector) {
const beforeResult = await getDOM(browser, resultSelector)
const beforeCount = beforeResult.count
await interact(browser, 'click', clickSelector)
// Wait for effect - result should appear or change
const maxRetries = 3
for (let i = 0; i < maxRetries; i++) {
await new Promise(r => setTimeout(r, 100 * (i + 1)))
const afterResult = await getDOM(browser, resultSelector)
if (afterResult.count > beforeCount) {
return afterResult.elements[0] // Success - effect happened
}
}
throw new Error(`Click on ${clickSelector} had no effect on ${resultSelector}`)
}
const successMsg = await clickWithEffect(browser, '.submit', '.success-msg')
Problem: Element is visible and clickable, but CSS animations are in progress; clicking during animation causes race conditions or intermediate states.
Example Fix:
// WRONG: Click element as soon as it's visible
await interact(browser, 'click', '.modal')
// RIGHT: Wait for animation to complete
async function waitForAnimationComplete(browser, selector) {
const element = await getDOM(browser, selector)
if (element.count === 0) throw new Error(`${selector} not found`)
// Query twice to check if position is stable
const rect1 = element.elements[0].rect
await new Promise(r => setTimeout(r, 100))
const element2 = await getDOM(browser, selector)
const rect2 = element2.elements[0].rect
// If rects differ significantly, animation is still playing
if (Math.abs(rect1.y - rect2.y) > 5 || Math.abs(rect1.x - rect2.x) > 5) {
// Animation in progress, wait longer
await new Promise(r => setTimeout(r, 300))
}
}
await waitForAnimationComplete(browser, '.modal')
await interact(browser, 'click', '.modal-button')
Common Errors:
| Error | Cause | Recovery |
|---|---|---|
Protocol error | Chrome crashed | Restart; relaunch browser |
Element not found | Selector invalid/timing | Adjust selector; increase wait |
Element not visible | Hidden/off-screen | Check CSS/DOM state; scroll |
Navigation timeout | Page slow/unresponsive | Increase timeout; check URL |
Connection closed | Browser terminated | Relaunch; check resources |
Best Practices:
success field before assertionsnavigate() wait conditions; don't rush actionsProper waiting is the foundation of non-flaky tests. Understanding the difference between states prevents races and timeouts.
The navigate() function supports different wait conditions:
Waits for DOM ready and page load event, but does NOT wait for network idle.
// Document ready but API calls still pending
const nav = await navigate(browser, 'http://example.com', { wait_for: 'document_ready' })
// getDOM() might return empty if data hasn't loaded yet
Waits for DOM ready + network idle (2 seconds with no pending requests).
// Recommended for most tests
const nav = await navigate(browser, 'http://example.com') // network_idle by default
// Page has loaded AND initial API calls completed
Waits for DOM ready + specific element to become visible.
// Wait for form before continuing
const nav = await navigate(browser, 'http://localhost:3001/login', {
wait_for: 'selector_visible',
selector: 'form[data-testid=login-form]'
})
if (!nav.loaded) {
throw new Error('Login form never appeared')
}
These are distinct states. Understand the differences:
Element is in the DOM tree.
<div style="display: none">Hidden button</div> exists but is not visiblegetDOM() returns count > 0, but may have visible: falseconst result = await getDOM(browser, '#hidden-button')
console.log(result.count) // 1 - element exists
console.log(result.elements[0].visible) // false - but not visible
Element is in DOM, rendered with non-zero dimensions, and CSS does not hide it.
getDOM() returns visible: true, but bounding rect may be off-screenconst result = await getDOM(browser, '#button-below-fold')
console.log(result.elements[0].visible) // true - rendered
console.log(result.elements[0].rect.y) // 2000 - below viewport (1080px)
Element is visible + in viewport + event handler is attached + not disabled.
pointer-events: none + not disabled attributeinteract(browser, 'click', selector) before checking return value// Element is visible but not yet clickable
const result = await getDOM(browser, '.submit-btn')
console.log(result.elements[0].visible) // true - visible
const clickResult = await interact(browser, 'click', '.submit-btn')
console.log(clickResult.success) // false - handler not attached yet, or disabled
console.log(clickResult.elementVisible) // true - was visible, but interaction failed
For robust tests, poll element state until it reaches desired condition or timeout:
async function waitForElement(browser, selector, maxWaitMs = 5000) {
const startTime = Date.now()
const pollIntervalMs = 100
while (Date.now() - startTime < maxWaitMs) {
const result = await getDOM(browser, selector)
if (result.count > 0 && result.elements[0].visible) {
return result.elements[0]
}
await new Promise(r => setTimeout(r, pollIntervalMs))
}
throw new Error(`Timeout waiting for ${selector} after ${maxWaitMs}ms`)
}
// Usage
const element = await waitForElement(browser, '.success-message')
async function waitForElementWithBackoff(browser, selector, maxWaitMs = 5000) {
const startTime = Date.now()
let pollIntervalMs = 50
while (Date.now() - startTime < maxWaitMs) {
const result = await getDOM(browser, selector)
if (result.count > 0 && result.elements[0].visible) {
return result.elements[0]
}
await new Promise(r => setTimeout(r, pollIntervalMs))
// Increase interval each iteration, cap at 500ms
pollIntervalMs = Math.min(pollIntervalMs * 1.5, 500)
}
throw new Error(`Timeout waiting for ${selector} after ${maxWaitMs}ms`)
}
async function waitForCondition(browser, condition, maxWaitMs = 5000) {
const startTime = Date.now()
while (Date.now() - startTime < maxWaitMs) {
if (await condition(browser)) {
return true
}
await new Promise(r => setTimeout(r, 100))
}
throw new Error(`Condition not met after ${maxWaitMs}ms`)
}
// Usage: Wait for error count to be zero
await waitForCondition(
browser,
async (b) => {
const errors = await getDOM(b, '.error-message')
return errors.count === 0
},
3000
)
async function waitForStableElement(browser, selector, maxWaitMs = 5000) {
const startTime = Date.now()
let lastRect = null
let stableCount = 0 // Need 2 consecutive identical rects
while (Date.now() - startTime < maxWaitMs) {
const result = await getDOM(browser, selector)
if (result.count === 0) {
lastRect = null
stableCount = 0
await new Promise(r => setTimeout(r, 100))
continue
}
const currentRect = result.elements[0].rect
// Check if rect hasn't changed
if (lastRect &&
lastRect.x === currentRect.x &&
lastRect.y === currentRect.y &&
lastRect.width === currentRect.width &&
lastRect.height === currentRect.height) {
stableCount++
if (stableCount >= 2) { // Stable for 200ms
return result.elements[0]
}
} else {
stableCount = 0 // Reset, element is still moving
}
lastRect = currentRect
await new Promise(r => setTimeout(r, 100))
}
throw new Error(`Element ${selector} never stabilized after ${maxWaitMs}ms`)
}
// Usage: Wait for animated modal to finish animating
const modal = await waitForStableElement(browser, '.modal', 3000)
Screenshots are powerful tools for debugging and documentation, but must be used correctly. They capture state, not truth.
Take a screenshot right before making an assertion. If assertion fails, you already have visual evidence.
// GOOD: Screenshot before assertion
const result = await getDOM(browser, '.success-message')
const shot = await screenshot(browser, 'before-success-check.png')
if (result.count === 0) {
// Screenshot already captured, shows why assertion failed
throw new Error('Success message not found - see before-success-check.png')
}
Capture page state when errors occur. This is the single most important debugging tool.
try {
await interact(browser, 'click', '.delete-button')
const confirmation = await getDOM(browser, '.confirmation-dialog')
if (confirmation.count === 0) {
throw new Error('Confirmation dialog did not appear')
}
} catch (err) {
// Capture page state at moment of failure
await screenshot(browser, `failure-${Date.now()}.png`)
throw err
}
Take screenshots at key success milestones. These serve as visual proof of feature working.
// Screenshot successful login
const loginSuccess = await getDOM(browser, '.dashboard-header')
if (loginSuccess.count > 0) {
await screenshot(browser, 'login-successful.png')
console.log('Login verified, screenshot saved')
}
Screenshots reveal issues that assertions can't detect:
// Before and after screenshots to detect layout/styling changes
await screenshot(browser, 'page-before-update.png')
await interact(browser, 'click', '.theme-toggle')
await new Promise(r => setTimeout(r, 300)) // Wait for theme animation
await screenshot(browser, 'page-after-update.png')
// Visual diff these screenshots to find CSS differences
// Screenshot shows if element is actually off-screen or covered
const element = await getDOM(browser, '.modal')
if (element.count > 0 && element.elements[0].visible) {
// getDOM says visible, but screenshot might show it's covered by overlay
const shot = await screenshot(browser, 'modal-overlap-check.png')
// Manual inspection: is modal actually visible, or hidden behind something?
}
// Screenshot catches CSS not applying, images not loading, etc.
await navigate(browser, 'http://localhost:3001/gallery')
// Images should be visible by now
const images = await getDOM(browser, 'img')
if (images.count === 0) {
// Why no images? Screenshot will show if page is blank or partially rendered
await screenshot(browser, 'gallery-broken.png')
throw new Error('Gallery images not found')
}
// Captures entire viewport (1920x1080)
const shot = await screenshot(browser, 'full-page.png')
// Good for: seeing overall layout, context around failure
For element-specific details, use getDOM() to get bounding rect, then crop screenshot:
// Get element's bounding box
const result = await getDOM(browser, '.button-to-crop')
const rect = result.elements[0].rect
console.log(`Crop: x=${rect.x}, y=${rect.y}, w=${rect.width}, h=${rect.height}`)
// Take full screenshot, then crop in post-processing or use external tool
const shot = await screenshot(browser, 'full-page.png')
// Use ImageMagick: convert full-page.png -crop 200x50+100+200 button.png
A test that passes sometimes and fails sometimes is flaky. Flakiness indicates a real problem, not randomness.
// Run same test 5 times, watch for intermittent failures
// If test passes sometimes and fails sometimes, it's flaky
// Root cause: Timing-dependent code
await interact(browser, 'click', '.save-button')
// What if handler takes 200ms to attach? Sometimes it's ready, sometimes not
const result = await getDOM(browser, '.success-message')
if (result.count === 0) {
throw new Error('Save failed') // Intermittent failure
}
// Test fails with --timeout 5000 but passes with --timeout 10000
// Clear sign: your test is waiting for something async, but timeout too short
// Bad: Fixed sleep (works on developer machine, fails in CI)
await interact(browser, 'click', '.submit')
await new Promise(r => setTimeout(r, 100)) // Works locally (maybe), fails on slow CI
const result = await getDOM(browser, '.result')
// Good: Polling with reasonable timeout
const element = await waitForElement(browser, '.result', 5000)
// This almost always indicates:
// 1. Timing assumption (element loads in 100ms on your machine, takes 2s in CI)
// 2. Viewport difference (your screen size != CI viewport)
// 3. Network difference (local loopback is fast, CI network is slow)
// Test that passes locally might miss this timing issue:
const result = await getDOM(browser, 'input[name=email]')
// input may not exist yet if API call takes 1.5s
// Fix: Explicit wait handles both fast and slow environments
const input = await waitForElement(browser, 'input[name=email]', 5000)
When a selector is intermittently unreliable, build in retry logic:
async function interactWithRetry(browser, action, selector, value, maxRetries = 3) {
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
const result = await interact(browser, action, selector, value)
if (result.success && result.elementVisible) {
return result // Success
}
if (attempt < maxRetries - 1) {
// Not yet successful, retry
console.log(`Attempt ${attempt + 1} failed, retrying...`)
await new Promise(r => setTimeout(r, Math.pow(2, attempt) * 100)) // Exponential backoff
continue
}
} catch (err) {
if (attempt < maxRetries - 1) {
console.log(`Attempt ${attempt + 1} threw error, retrying...`)
await new Promise(r => setTimeout(r, Math.pow(2, attempt) * 100))
continue
}
throw err
}
}
throw new Error(`Failed after ${maxRetries} attempts: ${selector}`)
}
// Usage
await interactWithRetry(browser, 'click', '.flaky-button')
When you discover a flaky test, don't just add delays. Diagnose the root cause:
Step 1: Gather Evidence
// Log element state before action
const beforeClick = await getDOM(browser, '.button')
console.log('Before click:', {
found: beforeClick.count > 0,
visible: beforeClick.elements[0]?.visible,
rect: beforeClick.elements[0]?.rect,
classes: beforeClick.elements[0]?.classList
})
// Try action and capture result
const clickResult = await interact(browser, 'click', '.button')
console.log('Click result:', clickResult)
// Check state after action
const afterClick = await getDOM(browser, '.result')
console.log('After click:', {
resultCount: afterClick.count,
resultVisible: afterClick.elements[0]?.visible
})
Step 2: Identify the Bottleneck
// Too-tight timeout (element takes 500ms to appear, timeout is 200ms)
// FIX: Increase timeout or use polling
const elem = await waitForElement(browser, '.result', 5000)
// Element appears then disappears (modal closes automatically)
// FIX: Capture value immediately after appearance
const modal = await getDOM(browser, '.modal')
const modalText = modal.elements[0]?.text // Capture immediately
// Then assert the value
// Animation in progress when clicking
// FIX: Wait for animation to complete before clicking
await waitForStableElement(browser, '.animated-button')
await interact(browser, 'click', '.animated-button')
// Handler not attached yet
// FIX: Retry click action with backoff
await interactWithRetry(browser, 'click', '.button-with-late-handler', null, 3)
Step 3: Implement Targeted Fix
// Once you identify the root cause, implement specific fix:
// Issue: Animation delay
// Fix: Wait for element to stabilize before interaction
async function clickAnimatedElement(browser, selector) {
await waitForStableElement(browser, selector, 2000)
return await interact(browser, 'click', selector)
}
// Issue: Async data loading
// Fix: Poll for element to appear with reasonable timeout
async function clickAfterDataLoads(browser, selector) {
await waitForElement(browser, selector, 5000)
return await interact(browser, 'click', selector)
}
// Issue: Handler attachment race
// Fix: Retry with exponential backoff
async function clickWithHandlerRetry(browser, selector) {
return await interactWithRetry(browser, 'click', selector, null, 3)
}
Step 4: Validate Fix
// Run test multiple times to confirm flakiness is gone
// Local: npm test -- --repeat=10
// CI: Run same test 10 times in pipeline, verify no intermittent failures
// Complete eval script example
const { launch, navigate, interact, screenshot, getDOM, teardown } = require('./eval-driver-web-cdp')
async function runEval() {
let browser
try {
// 1. Launch browser
browser = await launch()
console.log('Browser launched')
// 2. Navigate to login page
const nav = await navigate(browser, 'http://localhost:3001/login')
if (!nav.loaded) {
throw new Error(`Failed to load login page: ${nav.statusCode}`)
}
// 3. Check login form visible
const form = await getDOM(browser, 'form[data-testid=login-form]')
if (form.count === 0) {
throw new Error('Login form not found')
}
// 4. Interact with form
await interact(browser, 'type', 'input[name=email]', 'user@example.com')
await interact(browser, 'type', 'input[name=password]', 'correct-password')
await interact(browser, 'click', 'button[type=submit]')
// 5. Wait for navigation (re-navigate to capture new state)
const afterSubmit = await navigate(browser, 'http://localhost:3001/dashboard')
if (!afterSubmit.loaded) {
throw new Error('Dashboard failed to load after login')
}
// 6. Verify success
const success = await getDOM(browser, '.welcome-message')
if (success.count === 0) {
throw new Error('Success message not visible')
}
console.log('Login successful:', success.elements[0].text)
// 7. Screenshot for documentation
const shot = await screenshot(browser, 'dashboard.png')
console.log('Screenshot:', shot.path)
return { success: true, result: success.elements[0].text }
} catch (err) {
console.error('Eval failed:', err.message)
if (browser) {
await screenshot(browser, 'error-state.png')
}
throw err
} finally {
// 8. Always cleanup
if (browser) {
await teardown(browser)
}
}
}
// Run
runEval().then(() => {
console.log('Eval complete')
process.exit(0)
}).catch(err => {
console.error(err)
process.exit(1)
})
puppeteer, chrome-remote-interface, or raw WebSocket)chrome-remote-interface@^0.32.0 (or puppeteer/other CDP client)
| Term | Definition |
|---|---|
| CDP | Chrome DevTools Protocol - low-level API for Chrome automation |
| Headless | Browser runs without GUI, suitable for automation/testing |
| Viewport | Visible page area (1920x1080 default) |
| Network idle | No pending network requests for 2+ seconds |
| DOM | Document Object Model - in-memory representation of HTML |
| Selector | CSS query to find elements (e.g., #id, .class, [attr=value]) |
| Visibility | Element rendered and not hidden by CSS |
Future Enhancements:
Before running a CDP eval scenario:
launch() called with explicit viewport dimensions matching target device classnavigate() followed by explicit wait for load state (networkIdle or DOMContentLoaded)data-testid, ARIA role, or stable aria-label (not CSS class selectors)screenshot() called and file path recorded in scenario outputteardown() called in all paths (success, failure, timeout)qa/semantic-eval-run.log (no silent skips).PASS, FAIL, BLOCKED_DEPENDENCY, SKIPPED (with reason), CONTEXT_GAP — no unclassified results.qa/semantic-eval-manifest.json written with kind: semantic-csv-eval and a non-placeholder outcome.python3 tools/verify/verify_forge_task.py --task-id <id> --brain <brain> exits 0.| Skill / Doc | Relationship |
|---|---|
qa-semantic-csv-orchestrate | Dispatcher — invokes this driver for steps with Surface: web or web-cdp |
eval-judge | Downstream — reads semantic-eval-run.log entries this driver writes |
forge-eval-gate | Gate — this driver is one of multiple drivers coordinated by the gate |
docs/semantic-eval-csv.md | Surface → driver mapping; DependsOn syntax |
docs/semantic-eval-schema.md | semantic-eval-run.log outcome enum and required fields |
WHEN: You need to detect performance regressions before merge — TTFB, response time, bundle size. Run as part of forge-eval-gate or standalone before raising a PR.
WHEN: A decision is being superseded, deprecated, or has aged out. Archive it without deletion — marks as warm→cold→archived with full audit trail.
WHEN: You are writing a new decision, superseding an old one, or querying relationships across decisions. Create semantic edges between decisions and link concepts across products/projects/time.
WHEN: You are about to make a decision and need to check if prior art or past learnings exist. Recall decisions, patterns, and gotchas from the brain before proceeding.
WHEN: You need to trace the full provenance of a specific decision — who made it, when, why, and what alternatives were considered. Shows why, when, by whom, evidence, alternatives, outcome.
WHEN: You need to record a decision, lock a spec, log an eval run, or document learnings in the brain.