Production-grade browser automation using Playwright. Covers end-to-end testing, web scraping, synthetic monitoring, form automation, and screenshot capture — with safe defaults, robust selectors, and CI/CD portability.
Browser testing — "write an e2e test for the login flow", "test this form", "check if the dashboard loads", "automate browser regression tests"
Web scraping — "extract all product prices from this page", "scrape the table data", "get the article text", "crawl product listings from this site"
Form automation — "fill out this multi-step form", "submit the registration", "bulk-upload via the web UI", "automate this checkout flow"
Screenshot capture — "take a screenshot of the page", "capture the error state", "full-page screenshot of this blog", "screenshot every page of this site"
Synthetic monitoring — "check if the site is up and the login works", "monitor this checkout flow every 5 minutes", "set up health-check for the dashboard"
Multi-page flows — "go through the onboarding wizard", "walk through the purchase funnel", "verify the password-reset flow", "test the entire signup-to-purchase journey"
Do NOT trigger for:
Asking about browser features without automation intent ("what browsers support WebGPU?")
General Playwright API questions without a concrete task ("how does page.waitForSelector work?")
Discussing browser compatibility in the abstract
Requests to manually test something in a browser
UI/UX design feedback without automation
Asking "what's different between Chrome and Firefox rendering?" — factual, no automation
Mixing sync and async Playwright APIs in the same script
Pick one API and stay consistent. sync_playwright() for scripts, async_playwright() for test suites.
Forgetting --no-sandbox in Docker/CI
Add args=["--no-sandbox"] to every launch() call. Without it, Chromium refuses to start.
Using page.content() for data extraction instead of .evaluate_all() or .text_content()
page.content() returns raw HTML that you then have to parse. Use Playwright's built-in extraction.
Not handling cookie banners or modals before interacting with page content
Always dismiss cookie consents, accept dialogs, or close overlays before interacting.
Leaving browser processes open on script error
Always use context managers (async with / with blocks) — they clean up even on exceptions.
✅ Debugging Checklist (when things go wrong)
Did you wait for the element to be visible before interacting?
Is the selector valid? Test with playwright codegen to verify.
Are you using the right wait_until strategy for your page type (SPA vs MPA)?
If in CI/Docker, did you add --no-sandbox?
Is there a cookie consent modal blocking interaction?
Are you behind a proxy/VPN that interferes with browser network?
Did the page trigger a download dialog? Handle with page.on("download").
Is the browser closed too early? Check finally block or context manager exit order.
Workflow
Follow this ordered pipeline for every browser automation task:
1. Setup
# Synchronous (preferred for simple scripts)from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
context = browser.new_context(
viewport={"width": 1280, "height": 720},
user_agent="Mozilla/5.0 (compatible; AutomationBot/1.0)"
)
page = context.new_page()
# Async (required for pytest-playwright, larger suites)import asyncio
from playwright.async_api import async_playwright
asyncdefmain():
asyncwith async_playwright() as p:
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
asyncio.run(main())
Decision points:
Headless vs headed: Use headless=True by default. Use headless=False when the user needs to observe the action or debug a visual issue.
Sync vs async: Use sync for one-off scripts and quick tasks. Use async when writing pytest fixtures, concurrent scrapers, or large test suites.
Chromium vs Firefox vs WebKit: Default to Chromium for broadest compatibility. Use Firefox/WebKit only when explicitly requested for cross-browser testing.
CI tip: If the user mentions CI/CD, add args=["--no-sandbox"] to launch() for Docker/Linux environments.
2. Navigation
# Basic navigation with timeouttry:
page.goto("https://example.com", wait_until="domcontentloaded", timeout=30000)
except playwright._impl._api_types.TimeoutError:
print("Navigation timed out — site may be down or slow")
raise# Wait for network idle (SPA-heavy pages)
page.goto("https://spa-app.example.com", wait_until="networkidle")
# Useful post-navigation waits
page.wait_for_load_state("domcontentloaded") # HTML parsed
page.wait_for_load_state("load") # all resources loaded
page.wait_for_load_state("networkidle") # no network for 500ms
# Never use time.sleep(). Use these instead:await page.wait_for_selector("[data-testid='result']", state="visible", timeout=10000)
await page.wait_for_function("() => document.querySelector('.spinner') === null")
await page.wait_for_url("**/dashboard**")
await page.wait_for_load_state("networkidle")
# For dynamic content that appears/disappearsawait expect(page.get_by_text("Loading...")).to_be_hidden(timeout=15000)
await expect(page.get_by_text("Results")).to_be_visible(timeout=15000)
# For network-triggered updatesasyncwith page.expect_response(lambda r: "/api/results"in r.url):
await page.click("[data-testid='search-button']")
4. Assertion
from playwright.async_api import expect
# Page-levelawait expect(page).to_have_title("Dashboard — My App")
await expect(page).to_have_url("https://app.example.com/dashboard")
# Element visibilityawait expect(page.get_by_text("Welcome back")).to_be_visible()
await expect(page.locator(".error-banner")).to_be_hidden()
# Contentawait expect(page.get_by_test_id("user-name")).to_have_text("John Doe")
await expect(page.get_by_test_id("item-count")).to_contain_text("5")
# Form stateawait expect(page.get_by_label("Email")).to_have_value("user@example.com")
await expect(page.get_by_label("Agree")).to_be_checked()
# Screenshot-based verification
screenshot = await page.screenshot(full_page=True)
# For visual regression, combine with pixelmatch or Percy# Custom assertions for scraping
items = await page.locator(".product-card").count()
assert items >= 10, f"Expected at least 10 products, found {items}"
Assertion retry behavior: Playwright expect auto-retries for up to 5 seconds (configurable). This is usually what you want — it handles async rendering without brittle sleeps.
5. Cleanup
# With context managers (recommended)asyncwith async_playwright() as p:
asyncwithawait p.chromium.launch() as browser:
asyncwithawait browser.new_page() as page:
await page.goto("https://example.com")
# ... work ...# Everything auto-closes at block exit# Manual cleanup (when not using context managers)await page.close()
await context.close()
await browser.close()
await p.stop() # playwright instance
Always clean up. Orphaned browser processes leak memory and ports. Context managers are the safest default — they handle cleanup even on exceptions.
Error Handling
import asyncio
from playwright.async_api import async_playwright, TimeoutError as PlaywrightTimeout
asyncdefrobust_navigation(url: str, retries: int = 2):
"""Navigate with retry logic for flaky networks."""for attempt inrange(retries + 1):
try:
await page.goto(url, wait_until="domcontentloaded", timeout=15000)
returnexcept PlaywrightTimeout:
if attempt == retries:
raiseprint(f"Navigation attempt {attempt + 1} failed, retrying...")
await asyncio.sleep(2 ** attempt) # exponential backoff# Stale element recoverytry:
await page.click("[data-testid='dynamic-button']")
except PlaywrightTimeout:
# Element may have been removed and re-renderedawait page.wait_for_selector("[data-testid='dynamic-button']", state="attached")
await page.click("[data-testid='dynamic-button']")
# Network failure handlingtry:
await page.goto("https://flaky-service.example.com")
except Exception as e:
if"net::ERR_"instr(e):
raise RuntimeError(f"Network error accessing page: {e}")
raise# Modal/dialog handling (accept before interaction)
page.on("dialog", lambda dialog: dialog.accept())
# Mock API responses for stable testsawait page.route("**/api/users/**", lambda route: route.fulfill(
status=200,
content_type="application/json",
body='{"id": 1, "name": "Test User"}'
))
# Capture network requests for debugging
requests = []
page.on("request", lambda req: requests.append(f"{req.method}{req.url}"))
page.on("response", lambda res: print(f"{res.status}{res.url}"))
# Wait for specific API call to completeasyncwith page.expect_response(lambda r: "/api/submit"in r.url) as response_info:
await page.click("[data-testid='submit']")
response = await response_info.value
assert response.status == 200
Visual Regression (Screenshot Diffing)
# Capture and compare screenshotsawait page.screenshot(path="baseline.png", full_page=True)
# Use with pixelmatch, Percy, or Chromatic for automated diffing# Element-level screenshotawait page.locator(".pricing-table").screenshot(path="pricing.png")
# Clip to a specific region (avoid dynamic content)await page.screenshot(
path="header.png",
clip={"x": 0, "y": 0, "width": 1280, "height": 200}
)
Platform Compatibility Notes
Claude Code (VS Code / CLI)
Sync API preferred for quick scripts
Use subprocess.run(["python", "script.py"]) to execute
