Production-proven Playwright web scraping patterns with selector-first approach and robust error handling.
Use when users need to build web scrapers, extract data from websites, automate browser interactions,
or ask about Playwright selectors, text extraction (innerText vs textContent), regex patterns for HTML,
fallback hierarchies, or scraping best practices.
Production-proven Playwright web scraping patterns with selector-first approach and robust error handling.
Use when users need to build web scrapers, extract data from websites, automate browser interactions,
or ask about Playwright selectors, text extraction (innerText vs textContent), regex patterns for HTML,
fallback hierarchies, or scraping best practices.
Playwright Web Scraper
Production-proven web scraping patterns using Playwright with selector-first approach and robust error handling.
Core Principles
1. Selector-First Approach
Always prefer semantic locators over CSS selectors:
Critical difference between textContent and innerText:
// ❌ WRONG: Returns ALL text including hidden elements, scripts, iframesconst pageText = await page.textContent("body");
// ✅ CORRECT: Returns only VISIBLE text (what users see)const pageText = await page.innerText("body");
Use case for each:
innerText("body") - Extract visible content for regex matching
textContent(selector) - Get text from specific elements
3. Regex Patterns for Extraction
Handle newlines and whitespace in HTML:
// ❌ FAILS: [^$]* doesn't match across newlinesconst match = pageText.match(/ADULT[^$]*(\$\d+\.\d{2})/);
// ✅ WORKS: [\s\S]{0,10} matches any character including newlinesconst match = pageText.match(/ADULT[\s\S]{0,10}(\$\d+\.\d{2})/);
Use the Chrome DevTools MCP server to inspect actual page structure:
// In your conversation with Claude:// "Use Chrome DevTools to inspect the pricing page"// Claude will use: take_snapshot, evaluate_script, etc.
Logging Selectors
Always track which selectors worked:
constselectorsUsed: Record<string, string> = {};
// After each extraction
selectorsUsed.fieldName = "getByRole" | "regex" | "fallback-1";
// Return in response for debuggingreturn { data, selectorsUsed };
Visual Debugging
// Take screenshot at key pointsawait page.screenshot({ path: 'debug-step-1.png' });
// Highlight element before extractionawait page.locator(selector).highlight();
Anti-Patterns to Avoid
❌ Using hypothetical attributes
// DON'T assume data attributes existawait page.locator('[data-price]'); // Might not exist!
❌ Over-relying on CSS classes
// DON'T use implementation-specific classesawait page.locator('.MuiButton-root-xyz'); // Will break when CSS changes
❌ Ignoring visible vs. hidden text
// DON'T use textContent for regex extractionconst text = await page.textContent("body"); // Includes hidden iframes!
❌ Not handling missing data
// DON'T assume data existsconst price = await page.locator('.price').textContent(); // Might throw!// DO use optional chaining and null returnsconst price = await page.locator('.price').textContent().catch(() =>null);
