stealth-browser

name	stealth-browser
description	Anti-bot browser automation using Camoufox and Nodriver. Bypasses Cloudflare Turnstile, Datadome, and aggressive anti-bot on sites like Airbnb and Yelp. Use when standard Playwright/Selenium gets blocked.
metadata	{"openclaw":{"emoji":"🥷","requires":{"bins":["distrobox"],"env":[]}}}

Stealth Browser Skill 🥷

Anti-bot browser automation that bypasses Cloudflare Turnstile, Datadome, and aggressive fingerprinting.

When to Use

• Standard Playwright/Selenium gets blocked
• Site shows Cloudflare challenge or "checking your browser"
• Need to scrape Airbnb, Yelp, or similar protected sites
• playwright-stealth isn't working anymore

Tool Selection

Target Difficulty	Tool	When to Use
Browser	Camoufox	All protected sites - Cloudflare, Datadome, Yelp, Airbnb
API Only	curl_cffi	No browser needed, just TLS spoofing

Quick Start

All scripts run in pybox distrobox for isolation.

⚠️ Use python3.14 explicitly - pybox may have multiple Python versions with different packages installed.

1. Setup (First Time)

# Install tools in pybox (use python3.14)
distrobox-enter pybox -- python3.14 -m pip install camoufox curl_cffi

# Camoufox browser downloads automatically on first run (~700MB Firefox fork)

2. Fetch a Protected Page

Browser (Camoufox):

distrobox-enter pybox -- python3.14 scripts/camoufox-fetch.py "https://example.com" --headless

API only (curl_cffi):

distrobox-enter pybox -- python3.14 scripts/curl-api.py "https://api.example.com/endpoint"

Architecture

┌─────────────────────────────────────────────────────────┐
│                     OpenClaw Agent                       │
├─────────────────────────────────────────────────────────┤
│  distrobox-enter pybox -- python3.14 scripts/xxx.py         │
├─────────────────────────────────────────────────────────┤
│                      pybox Container                     │
│         ┌─────────────┐  ┌─────────────┐               │
│         │  Camoufox   │  │  curl_cffi  │               │
│         │  (Firefox)  │  │  (TLS spoof)│               │
│         └─────────────┘  └─────────────┘               │
└─────────────────────────────────────────────────────────┘

Tool Details

Camoufox

• What: Custom Firefox build with C++ level stealth patches
• Pros: Best fingerprint evasion, passes Turnstile automatically
• Cons: ~700MB download, Firefox-based
• Best for: All protected sites - Cloudflare, Datadome, Yelp, Airbnb

curl_cffi

• What: Python HTTP client with browser TLS fingerprint spoofing
• Pros: No browser overhead, very fast
• Cons: No JS execution, API endpoints only
• Best for: Known API endpoints, mobile app reverse engineering

Critical: Proxy Requirements

Datacenter IPs (AWS, DigitalOcean) = INSTANT BLOCK on Airbnb/Yelp

You MUST use residential or mobile proxies:

# Example proxy config
proxy = "http://user:[e-mail]:8080"

See references/proxy-setup.md for proxy configuration.

Behavioral Tips

Sites like Airbnb/Yelp use behavioral analysis. To avoid detection:

1. Warm up: Don't hit target URL directly. Visit homepage first, scroll, click around.
2. Mouse movements: Inject random mouse movements (Camoufox handles this).
3. Timing: Add random delays (2-5s between actions), not fixed intervals.
4. Session stickiness: Use same proxy IP for 10-30 min sessions, don't rotate every request.

Headless Mode Warning

⚠️ Old --headless flag is DETECTED. Options:

1. New Headless: Use headless="new" (Chrome 109+)
2. Xvfb: Run headed browser in virtual display
3. Headed: Just run headed if you can (most reliable)

# Xvfb approach (Linux)
Xvfb :99 -screen 0 1920x1080x24 &
export DISPLAY=:99
python scripts/camoufox-fetch.py "https://example.com"

Troubleshooting

Problem	Solution
"Access Denied" immediately	Use residential proxy
Cloudflare challenge loops	Try Camoufox instead of Nodriver
Browser crashes in pybox	Install missing deps: sudo dnf install gtk3 libXt
TLS fingerprint blocked	Use curl_cffi with impersonate="chrome120"
Turnstile checkbox appears	Add mouse movement, increase wait time
ModuleNotFoundError: camoufox	Use python3.14 not python or python3
greenlet segfault (exit 139)	Python version mismatch - use python3.14 explicitly
libstdc++.so.6 errors	NixOS lib path issue - use python3.14 in pybox

Python Version Issues (NixOS/pybox)

The pybox container may have multiple Python versions with separate site-packages:

# Check which Python has camoufox
distrobox-enter pybox -- python3.14 -c "import camoufox; print('OK')"

# Wrong (may use different Python)
distrobox-enter pybox -- python3.14 scripts/camoufox-session.py ...

# Correct (explicit version)
distrobox-enter pybox -- python3.14 scripts/camoufox-session.py ...

If you get segfaults or import errors, always use python3.14 explicitly.

Examples

Scrape Airbnb Listing

distrobox-enter pybox -- python3.14 scripts/camoufox-fetch.py \
  "https://www.airbnb.com/rooms/12345" \
  --headless --wait 10 \
  --screenshot airbnb.png

Scrape Yelp Business

distrobox-enter pybox -- python3.14 scripts/camoufox-fetch.py \
  "https://www.yelp.com/biz/some-restaurant" \
  --headless --wait 8 \
  --output yelp.html

API Scraping with TLS Spoofing

distrobox-enter pybox -- python3.14 scripts/curl-api.py \
  "https://api.yelp.com/v3/businesses/search?term=coffee&location=SF" \
  --headers '{"Authorization": "Bearer xxx"}'

Session Management

Persistent sessions allow reusing authenticated state across runs without re-logging in.

Quick Start

# 1. Login interactively (headed browser opens)
distrobox-enter pybox -- python3.14 scripts/camoufox-session.py \
  --profile airbnb --login "https://www.airbnb.com/account-settings"

# Complete login in browser, then press Enter to save session

# 2. Reuse session in headless mode
distrobox-enter pybox -- python3.14 scripts/camoufox-session.py \
  --profile airbnb --headless "https://www.airbnb.com/trips"

# 3. Check session status
distrobox-enter pybox -- python3.14 scripts/camoufox-session.py \
  --profile airbnb --status "https://www.airbnb.com"

Flags

Flag	Description
--profile NAME	Named profile for session storage (required)
--login	Interactive login mode - opens headed browser
--headless	Use saved session in headless mode
--status	Check if session appears valid
--export-cookies FILE	Export cookies to JSON for backup
--import-cookies FILE	Import cookies from JSON file

Storage

• Location: ~/.stealth-browser/profiles/<name>/
• Permissions: Directory 700, files 600
• Profile names: Letters, numbers, _, - only (1-63 chars)

Cookie Handling

• Save: All cookies from all domains stored in browser profile
• Restore: Only cookies matching target URL domain are used
• SSO: If redirected to Google/auth domain, re-authenticate once and profile updates

Login Wall Detection

The script detects session expiry using multiple signals:

1. HTTP status: 401, 403
2. URL patterns: /login, /signin, /auth
3. Title patterns: "login", "sign in", etc.
4. Content keywords: "captcha", "verify", "authenticate"
5. Form detection: Password input fields

If detected during --headless mode, you'll see:

🔒 Login wall signals: url-path, password-form

Re-run with --login to refresh the session.

Remote Login (SSH)

Since --login requires a visible browser, you need display forwarding:

X11 Forwarding (Preferred):

# Connect with X11 forwarding
ssh -X user@server

# Run login (opens browser on your local machine)
distrobox-enter pybox -- python3.14 scripts/camoufox-session.py \
  --profile mysite --login "https://example.com"

VNC Alternative:

# On server: start VNC session
vncserver :1

# On client: connect to VNC
vncviewer server:1

# In VNC session: run login
distrobox-enter pybox -- python3.14 scripts/camoufox-session.py \
  --profile mysite --login "https://example.com"

Security Notes

⚠️ Cookies are credentials. Treat profile directories like passwords:

• Profile dirs have chmod 700 (owner only)
• Cookie exports have chmod 600
• Don't share profiles or exported cookies over insecure channels
• Consider encrypting backups

Limitations

Limitation	Reason
localStorage/sessionStorage not exported	Use browser profile instead (handles automatically)
IndexedDB not portable	Stored in browser profile, not cookie export
No parallel profile access	No file locking in v1; use one process per profile

References

• references/proxy-setup.md — Proxy configuration guide
• references/fingerprint-checks.md — What anti-bot systems check