cloud-browser-automation

# Cloud Browser Automation Use cloud browser services (Browserbase, Browser Use Cloud) for web scraping when local tools fail due to Cloudflare protection, JavaScript rendering requirements, or anti-bot measures. ## Usage Strategy **CRITICAL: On-demand only, close immediately** Always follow this pattern: 1. Create session ONLY when needed 2. Do the task (scrape/navigate/extract) 3. Close session IMMEDIATELY after done 4. Never leave sessions running idle Why: - Save credits (pay per minute) - Avoid hitting concurrent session limits (e.g., Browserbase max 3) - Clean resource usage - No waste ## When to Use **Escalation ladder** (always try cheapest/fastest first): ``` 1. Try Hermes web_fetch first ↓ (if fails or empty content) 2. Try direct curl with proper headers (Pattern 0 — works for most static sites) ↓ (if blocked or needs JS) 3. Try Hermes browser ↓ (if blocked by Cloudflare or sandbox issues) 4. Use cloud browser (Browserbase) ``` **Use cloud browser when:** - ✅ Cloudflare Turnstile blocking requests - ✅ JavaScript-heavy sites (SPA, dynamic content) - ✅ Anti-bot detection (fingerprinting, TLS checks) - ✅ Complex multi-step automation (login, navigate, extract) **DON'T use cloud browser when:** - ❌ Simple HTTP requests (use web_fetch) - ❌ Static content (use web_fetch) - ❌ Cost-sensitive tasks (cloud = pay per minute) ## Browserbase Setup ### API Configuration ```bash # Credentials (saved in memory) API_KEY="bb_live_O6tVgdzl8B6WquBSj1XpuaC5hMc" PROJECT_ID="a5008864-bbaa-4966-96e2-2272497b003d" BASE_URL="https://www.browserbase.com/v1" # Limits MAX_CONCURRENT_SESSIONS=3 SESSION_TIMEOUT=5 # minutes (default) ``` ### Session Management **CRITICAL USER PREFERENCE:** > "gini jadi lo pakai browserbase saat dibutuhkan aja, nah sesi langsung close saat beres" **Translation:** Use on-demand only, close immediately after done. **Why:** - ✅ Save credits (pay per minute) - ✅ Avoid hitting concurrent limit (max 3) - ✅ Clean resource usage - ✅ No waste **Pattern:** ```python # 1. Create session ONLY when needed session = create_session() # 2. Do the task data = scrape_page(session) # 3. Close IMMEDIATELY after done close_session(session) ``` **DON'T:** - ❌ Create session at start of script "just in case" - ❌ Leave sessions running between tasks - ❌ Reuse sessions across multiple unrelated tasks - ❌ Forget to close sessions ## Workflow ### Step 1: Create Session ```bash curl -X POST "https://www.browserbase.com/v1/sessions" \ -H "X-BB-API-Key: $API_KEY" \ -H "Content-Type: application/json" \ -d '{ "projectId": "'$PROJECT_ID'", "browserSettings": { "viewport": {"width": 1920, "height": 1080} } }' ``` **Response:** ```json { "id": "session-id-here", "status": "RUNNING", "connectUrl": "wss://connect-v2.usw2.browserbase.com/?signingKey=...", "seleniumRemoteUrl": "http://connect-v2.usw2.browserbase.com/webdriver", "expiresAt": "2026-05-03T13:00:00.000Z" } ``` ### Step 2: Automate (Requires SDK) **Option A: Playwright** (Python) ```python from playwright.async_api import async_playwright async with async_playwright() as p: browser = await p.chromium.connect_over_cdp(connect_url) page = await browser.new_page() await page.goto("https://example.com") data = await page.evaluate("() => document.body.innerText") await browser.close() ``` **Option B: Selenium** (Python) ```python from selenium import webdriver options = webdriver.ChromeOptions() driver = webdriver.Remote( command_executor=selenium_remote_url, options=options ) driver.get("https://example.com") data = driver.find_element(By.TAG_NAME, "body").text driver.quit() ``` **Option C: Manual (Fallback)** ```bash # Get debug URL curl "https://www.browserbase.com/v1/sessions/$SESSION_ID/debug" \ -H "X-BB-API-Key: $API_KEY" # Open debug URL in browser # Navigate manually # Screenshot/extract data manually ``` ### Step 3: Close Session ```bash curl -X POST "https://www.browserbase.com/v1/sessions/$SESSION_ID/stop" \ -H "X-BB-API-Key: $API_KEY" ``` **ALWAYS close, even if task fails:** ```python try: data = scrape_with_browserbase() finally: close_session() # Always close! ``` ## VPS Constraints **Problem:** Ubuntu 24.04 strict package management blocks `pip install playwright/selenium`. **Solutions:** ### Option 1: Use Docker (Recommended) ```bash # Run Playwright in container docker run -it mcr.microsoft.com/playwright/python:v1.40.0-jammy \ python3 /tmp/scraper.py ``` **Pros:** Isolated, no pip conflicts **Cons:** Need Docker (~500MB) ### Option 2: Manual Fallback ```bash # Create session SESSION_ID=$(curl -X POST ... | jq -r '.id') # Get debug URL DEBUG_URL=$(curl ... | jq -r '.debuggerFullscreenUrl') # User opens URL manually echo "Open: $DEBUG_URL" echo "Navigate to target site" echo "Screenshot and send to me" # Close session curl -X POST ".../sessions/$SESSION_ID/stop" ... ``` ### Option 3: Delegate to Subagent ```python # If automation blocked, delegate research delegate_task( goal="Research [topic] via web search", toolsets=["web", "terminal"] ) ``` ## Common Patterns ### Pattern 0: Direct Curl for Static Sites (No Browserbase) **Challenge:** Site returns HTML but might have Cloudflare or anti-bot checks **Solution:** ```bash # Direct curl with proper headers (often bypasses basic protection) curl -s -L -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" \ -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" \ -H "Accept-Language: en-US,en;q=0.5" \ -H "Connection: keep-alive" \ -H "Upgrade-Insecure-Requests: 1" \ "https://example.com" > /tmp/page.html # If response is gzip-encoded, check file type first file /tmp/page.html # Should show "HTML document" not "gzip compressed" # Extract data with grep/sed/python grep -oP '<title>\K[^<]+' /tmp/page.html ``` **When to use:** - ✅ Static HTML sites (marketing pages, landing pages) - ✅ Sites with basic Cloudflare (not Turnstile) - ✅ Public content (no login required) - ✅ VPS without Docker/Playwright **When to escalate:** - ❌ Cloudflare Turnstile challenge appears - ❌ JavaScript-heavy SPA (empty body in curl response) - ❌ Login-required content ### Pattern 0b: TinyFish Browser for Cloudflare Bypass (Recommended) **Challenge:** API returns Cloudflare challenge page, need full browser automation **Solution:** ```python import requests import asyncio import websockets import json import time TINYFISH_API_KEY = "your-api-key" async def fetch_via_tinyfish(url): # 1. Create browser session headers = {"X-API-Key": TINYFISH_API_KEY, "Content-Type": "application/json"} payload = { "url": url, "timeout_seconds": 300, "browser_profile": "stealth", "proxy_config": {"enabled": True, "type": "tetra", "country_code": "US"} } resp = requests.post("https://api.browser.tinyfish.ai", headers=headers, json=payload, timeout=60) info = resp.json() session_id = info["session_id"] base_url = info["base_url"].rstrip("/") # 2. Wait for Cloudflare bypass (CRITICAL: 30 attempts × 6 seconds) page_ws_url = None for attempt in range(30): # Not 20! time.sleep(6) # Not 5! try: pages_resp = requests.get(f"{base_url}/pages", timeout=20) pages = pages_resp.json() if pages: page = pages[0] title = page.get("title", "") url_check = page.get("url", "") # Check BOTH title AND url (not just title!) if "Just a moment" not in title and url in url_check: page_ws_url = page.get("webSocketDebuggerUrl") break except: pass if not page_ws_url: requests.delete(f"https://api.browser.tinyfish.ai/{session_id}", headers={"X-API-Key": TINYFISH_API_KEY}) raise Exception("Failed to bypass Cloudflare after 30 attempts") # 3. Execute via CDP async with websockets.connect(page_ws_url, max_size=20_000_000) as ws: counter = 0 async def send(method, params=None): nonlocal counter counter += 1 msg_id = counter await ws.send(json.dumps({"id": msg_id, "method": method, "params": params or {}})) while True: msg = json.loads(await ws.recv()) if msg.get("id") == msg_id: return msg await send("Runtime.enable") result = await send("Runtime.evaluate", { "expression": """ fetch('/api/endpoint') .then(r => r.json()) .catch(e => ({error: String(e)})) """, "awaitPromise": True, "returnByValue": True }) data = result.get("result", {}).get("result", {}).get("value") # 4. Close session immediately requests.delete(f"https://api.browser.tinyfish.ai/{session_id}", headers={"X-API-Key": TINYFISH_API_KEY}) return data # Usage data = asyncio.run(fetch_via_tinyfish("https://example.com")) ``` **CRITICAL Cloudflare Bypass Settings:** - **30 attempts** (not 20) — Cloudflare can take 2-3 minutes - **6 second delay** (not 5) — Faster polling causes false negatives - **Check both title AND url** — Title alone insufficient for SPA sites - **max_size=20_000_000** for websocket — Large responses need bigger buffer **When to use:** - ✅ Cloudflare Turnstile v2 (most reliable bypass) - ✅ JavaScript-heavy SPA (full browser context) - ✅ Login-required content (can handle auth flows) - ✅ Production monitoring (stable, proven) **When to escalate to Browserbase:** - ❌ Need persistent sessions (TinyFish auto-closes) - ❌ Need screenshots/video recording - ❌ Complex multi-page workflows (TinyFish better for single-page scrapes) **Real-world example:** RPOW2Swap API monitoring (see `references/rpow2swap-cloudflare-bypass.md`) **Pattern 0a: Analyzing Minified JS Bundles** If curl returns a SPA with minified JavaScript, you can often extract API patterns without Browserbase. See `references/minified-js-analysis.md` for grep-based techniques to find: - API endpoints and base URLs - Data structures (orderbook, pricing, etc.) - Authentication patterns (localStorage tokens, headers) - WebSocket URLs This works for most modern SPAs (React, Vue, Next.js) and costs zero resources vs Browserbase. ### Pattern 1: Twitter/X Scraping (No Browserbase) **Challenge:** Need to read public tweet content **Solution:** ```bash # Direct curl scraping (faster + free) curl -s -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" \ "https://x.com/USERNAME/status/TWEET_ID" > /tmp/tweet.html # Extract JSON data grep -o 'window.__INITIAL_STATE__={.*}' /tmp/tweet.html | \ sed 's/window.__INITIAL_STATE__=//' > /tmp/tweet.json # Parse with jq or Python jq '.entities.tweets.entities' /tmp/tweet.json ``` **When to use:** - ✅ Public tweets (no login required) - ✅ Simple text/media extraction - ✅ Cost-sensitive tasks **When to escalate to Browserbase:** - ❌ Rate limited (429 errors) - ❌ Login-required content - ❌ Need to interact (like, retweet, reply) See `references/twitter-scraping-fallback.md` for full technique. ### Pattern 1: OpenSea NFT Scraping **Challenge:** OpenSea blocks direct HTTP (403 Cloudflare)