stealth-browser

# Stealth Browser Automation Silent, undetectable web automation combining multiple anti-detection layers. ## Quick Login Workflow (IMPORTANT) When user asks to login to any website: 1. **Open in headed mode** (visible browser for manual login): ```bash python scripts/stealth_session.py -u "https://target.com/login" -s sitename --headed ``` 2. **User logs in manually** in the visible browser 3. **Save session** after login confirmed: ```bash python scripts/stealth_session.py -u "https://target.com" -s sitename --headed --save ``` 4. **Future use** - load saved session (headless): ```bash python scripts/stealth_session.py -u "https://target.com" -s sitename --load ``` Sessions stored in: `~/.clawdbot/browser-sessions/<sitename>.json` ## 执行策略 (IMPORTANT) ### 1. 先静默后显示 - 优先使用 headless 模式静默尝试 - 如果失败或需要验证码，再切换到 headed 显示模式 - 避免打扰用户操作 ### 2. 断点续传 长任务使用 `task_runner.py` 管理状态： ```python from task_runner import TaskRunner task = TaskRunner('my_task') task.set_total(100) for i in items: if task.is_completed(i): continue # 跳过已完成 # 处理... task.mark_completed(i) task.finish() ``` ### 3. 超时处理 - 默认单页超时: 30秒 - 长任务每50项保存一次进度 - 失败自动重试3次 ### 4. 记录尝试 所有登录尝试记录在: `~/.clawdbot/browser-sessions/attempts.json` ## Architecture ``` ┌─────────────────────────────────────────────────────┐ │ Stealth Browser │ ├─────────────────────────────────────────────────────┤ │ Layer 1: Anti-Detection Engine │ │ - puppeteer-extra-plugin-stealth │ │ - Browser fingerprint spoofing │ │ - WebGL/Canvas/Audio fingerprint masking │ ├─────────────────────────────────────────────────────┤ │ Layer 2: Challenge Bypass │ │ - Cloudflare Turnstile/JS Challenge │ │ - hCaptcha / reCAPTCHA integration │ │ - 2Captcha / Anti-Captcha API │ ├─────────────────────────────────────────────────────┤ │ Layer 3: Session Persistence │ │ - Cookie storage (JSON/SQLite) │ │ - localStorage sync │ │ - Multi-profile management │ ├─────────────────────────────────────────────────────┤ │ Layer 4: Proxy & Identity │ │ - Rotating residential proxies │ │ - User-Agent rotation │ │ - Timezone/Locale spoofing │ └─────────────────────────────────────────────────────┘ ``` ## Setup ### Install Core Dependencies ```bash npm install -g puppeteer-extra puppeteer-extra-plugin-stealth npm install -g playwright pip install undetected-chromedriver DrissionPage ``` ### Optional: CAPTCHA Solvers Store API keys in `~/.clawdbot/secrets/captcha.json`: ```json { "2captcha": "YOUR_2CAPTCHA_KEY", "anticaptcha": "YOUR_ANTICAPTCHA_KEY", "capsolver": "YOUR_CAPSOLVER_KEY" } ``` ### Optional: Proxy Configuration Store in `~/.clawdbot/secrets/proxies.json`: ```json { "rotating": "http://user:pass@proxy.provider.com:port", "residential": ["socks5://ip1:port", "socks5://ip2:port"], "datacenter": "http://dc-proxy:port" } ``` ## Quick Start ### 1. Stealth Session (Python - Recommended) ```python # scripts/stealth_session.py - use for maximum compatibility import undetected_chromedriver as uc from DrissionPage import ChromiumPage # Option A: undetected-chromedriver (Selenium-based) driver = uc.Chrome(headless=True, use_subprocess=True) driver.get("https://nowsecure.nl") # Test anti-detection # Option B: DrissionPage (faster, native Python) page = ChromiumPage() page.get("https://cloudflare-protected-site.com") ``` ### 2. Stealth Session (Node.js) ```javascript // scripts/stealth.mjs import puppeteer from 'puppeteer-extra'; import StealthPlugin from 'puppeteer-extra-plugin-stealth'; puppeteer.use(StealthPlugin()); const browser = await puppeteer.launch({ headless: 'new', args: [ '--disable-blink-features=AutomationControlled', '--disable-dev-shm-usage', '--no-sandbox' ] }); const page = await browser.newPage(); await page.goto('https://bot.sannysoft.com'); // Verify stealth ``` ## Core Operations ### Open Stealth Page ```bash # Using agent-browser with stealth profile agent-browser --profile ~/.stealth-profile open https://target.com # Or via script python scripts/stealth_open.py --url "https://target.com" --headless ``` ### Bypass Cloudflare ```python # Automatic CF bypass with DrissionPage from DrissionPage import ChromiumPage page = ChromiumPage() page.get("https://cloudflare-site.com") # DrissionPage waits for CF challenge automatically # Manual wait if needed page.wait.ele_displayed("main-content", timeout=30) ``` For stubborn Cloudflare sites, use FlareSolverr: ```bash # Start FlareSolverr container docker run -d --name flaresolverr -p 8191:8191 ghcr.io/flaresolverr/flaresolverr # Request clearance curl -X POST http://localhost:8191/v1 \ -H "Content-Type: application/json" \ -d '{"cmd":"request.get","url":"https://cf-protected.com","maxTimeout":60000}' ``` ### Solve CAPTCHAs ```python # scripts/solve_captcha.py import requests import json import time def solve_recaptcha(site_key, page_url, api_key): """Solve reCAPTCHA v2/v3 via 2Captcha""" # Submit task resp = requests.post("http://2captcha.com/in.php", data={ "key": api_key, "method": "userrecaptcha", "googlekey": site_key, "pageurl": page_url, "json": 1 }).json() task_id = resp["request"] # Poll for result for _ in range(60): time.sleep(3) result = requests.get(f"http://2captcha.com/res.php?key={api_key}&action=get&id={task_id}&json=1").json() if result["status"] == 1: return result["request"] # Token return None def solve_hcaptcha(site_key, page_url, api_key): """Solve hCaptcha via Anti-Captcha""" resp = requests.post("https://api.anti-captcha.com/createTask", json={ "clientKey": api_key, "task": { "type": "HCaptchaTaskProxyless", "websiteURL": page_url, "websiteKey": site_key } }).json() task_id = resp["taskId"] for _ in range(60): time.sleep(3) result = requests.post("https://api.anti-captcha.com/getTaskResult", json={ "clientKey": api_key, "taskId": task_id }).json() if result["status"] == "ready": return result["solution"]["gRecaptchaResponse"] return None ``` ### Persistent Sessions ```python # scripts/session_manager.py import json import os from pathlib import Path SESSIONS_DIR = Path.home() / ".clawdbot" / "browser-sessions" SESSIONS_DIR.mkdir(parents=True, exist_ok=True) def save_cookies(driver, session_name): """Save cookies to JSON""" cookies = driver.get_cookies() path = SESSIONS_DIR / f"{session_name}_cookies.json" path.write_text(json.dumps(cookies, indent=2)) return path def load_cookies(driver, session_name): """Load cookies from saved session""" path = SESSIONS_DIR / f"{session_name}_cookies.json" if path.exists(): cookies = json.loads(path.read_text()) for cookie in cookies: driver.add_cookie(cookie) return True return False def save_local_storage(page, session_name): """Save localStorage""" ls = page.evaluate("() => JSON.stringify(localStorage)") path = SESSIONS_DIR / f"{session_name}_localStorage.json" path.write_text(ls) return path def load_local_storage(page, session_name): """Restore localStorage""" path = SESSIONS_DIR / f"{session_name}_localStorage.json" if path.exists(): data = path.read_text() page.evaluate(f"(data) => {{ Object.entries(JSON.parse(data)).forEach(([k,v]) => localStorage.setItem(k,v)) }}", data) return True return False ``` ### Silent Automation Workflow ```python # Complete silent automation example from DrissionPage import ChromiumPage, ChromiumOptions # Configure for stealth options = ChromiumOptions() options.headless() options.set_argument('--disable-blink-features=AutomationControlled') options.set_argument('--disable-dev-shm-usage') options.set_user_agent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36') page = ChromiumPage(options) # Navigate with CF bypass page.get("https://target-site.com") # Wait for any challenges page.wait.doc_loaded() # Interact silently page.ele("@id=username").input("user@email.com") page.ele("@id=password").input("password123") page.ele("@type=submit").click() # Save session for reuse page.cookies.save("~/.clawdbot/browser-sessions/target-site.json") ``` ## Proxy Rotation ```python # scripts/proxy_rotate.py import random import json from pathlib import Path def get_proxy(): """Get random proxy from pool""" config = json.loads((Path.home() / ".clawdbot/secrets/proxies.json").read_text()) proxies = config.get("residential", []) return random.choice(proxies) if proxies else config.get("rotating") # Use with DrissionPage options = ChromiumOptions() options.set_proxy(get_proxy()) page = ChromiumPage(options) ``` ## User Input Required To complete this skill, provide: 1. **CAPTCHA API Keys** (optional but recommended): - 2Captcha key: https://2captcha.com - Anti-Captcha key: https://anti-captcha.com - CapSolver key: https://capsolver.com 2. **Proxy Configuration** (optional): - Residential proxy provider credentials - Or list of SOCKS5/HTTP proxies 3. **Target Sites** (for pre-configured sessions): - Which sites need login persistence? - What credentials should be stored? ## Files Structure ``` stealth-browser/ ├── SKILL.md ├── scripts/ │ ├── stealth_session.py # Main stealth browser wrapper │ ├── solve_captcha.py # CAPTCHA solving utilities │ ├── session_manager.py # Cookie/localStorage persistence │ ├── proxy_rotate.py # Proxy rotation │ └── cf_bypass.py # Cloudflare-specific bypass └── references/ ├── fingerprints.md # Browser fingerprint details └── detection-tests.md # Sites to test anti-detection ``` ## Testing Anti-Detection ```bash # Run these to verify stealth is working: python scripts/stealth_open.py --url "https://bot.sannysoft.com" python scripts/stealth_open.py --url "https://nowsecure.nl" python scripts/stealth_open.py --url "https://arh.antoinevastel.com/bots/areyouheadless" python scripts/stealth_open.py --url "https://pixelscan.net" ``` ## Integration with agent-browser For simple tasks, use agent-browser with a persistent profile: ```bash # Create stealth profile once agent-browser --profile ~/.stealth-profile --headed open https://login-site.com # Login manually, then close # Reuse authenticated session (headless) agent-browser --profile ~/.stealth-profile snapshot agent-browser --profile ~/.stealth-profile click @e5 ``` For Cloudflare or CAPTCHA-heavy sites, use Python scripts instead. ## Best Practices 1. **Always use headless: 'new'** not `headless: true` (less detectable) 2. **Rotate User-Agents** matching browser version 3. **Add random delays** between actions (100-500ms) 4. **Use residential proxies** for sensitive targets 5. **Save sessions** after successful login 6. **Test on bot.sannysoft.com** before production use