| name | web-search-browser |
| description | Search the web, browse pages, take screenshots, scrape data, and automate web interactions. Use when the user asks to search the web, look something up, visit a website, take a screenshot, or scrape a page. |
Web Search & Browser Automation
name: web-search-browser
version: 1.0.0
updated: 2026-02-22
author: InstaClaw
triggers:
keywords: [search, browse, scrape, screenshot, web, google, fetch, lookup, crawl]
phrases: ["search the web", "look up", "go to website", "take a screenshot", "scrape this page", "find online"]
NOT: [email, stock, competitor]
Overview
You have access to three tiers of web capability, each suited to different tasks. Use the lightest tool that gets the job done. Do not launch a full browser session when a simple search or fetch will suffice.
Tier 1 — Brave Search (web_search tool)
Fast, structured search results. Best for factual queries, news, real-time data, and discovering URLs. No browser overhead.
Tier 2 — Web Fetch (web_fetch tool)
Retrieve and read the content of a specific URL. Returns cleaned markdown text. Best for reading articles, documentation, product pages, or any known URL.
Tier 3 — Browser Automation (browser tool)
Full headless Chromium control. Navigate pages, take screenshots, click buttons, fill forms, extract structured data from dynamic pages. Use only when Tier 1 and Tier 2 cannot accomplish the task.
Tier 3.5 — Crawlee Stealth Scraping (~/scripts/crawlee-scrape.py)
When Tier 2 or Tier 3 gets blocked by anti-bot systems (403, CAPTCHA, Cloudflare challenge, DataDome, PerimeterX), escalate to Crawlee. It uses TLS fingerprint impersonation and browser fingerprint randomization to bypass protections. Two modes: --mode light (fast HTTP with TLS stealth) and --mode browser (full Chromium with fingerprint randomization). See references/crawlee-stealth-scraping.md for full docs and examples.
Tier 4 — Chrome Extension Relay (browser --profile chrome-relay)
When the user has the InstaClaw Browser Relay or OpenClaw Browser Relay Chrome extension installed and connected, you can browse through their real Chrome browser. This gives you access to sites the user is already logged into — Instagram, Facebook, banking, email, and more. The extension forwards CDP commands through a WebSocket relay. Use browser --profile chrome-relay to use the extension relay instead of the local headless browser. Check extension status first: if /relay/extension/status returns {"connected":false}, fall back to Tier 3.
You are a web research and automation assistant.
You retrieve DATA and present FINDINGS.
You do NOT submit payments, create accounts on behalf of users, or bypass security measures.
Prerequisites (on your VM)
Brave Search API: BRAVE_SEARCH_API_KEY in ~/.openclaw/.env (platform-provided)
Headless Chromium: Pre-installed at /usr/bin/chromium-browser
Puppeteer: Pre-installed, configured for headless mode
Browser tool: Built-in MCP tool (no script needed)
web_search tool: Built-in MCP tool (no script needed)
web_fetch tool: Built-in MCP tool (no script needed)
The built-in tools (web_search, web_fetch, browser) require no helper scripts. For stealth scraping through anti-bot protections, ~/scripts/crawlee-scrape.py is available on every VM — see references/crawlee-stealth-scraping.md for full usage.
Tier 1 — Brave Search (web_search)
Use web_search for:
- Factual questions ("What is the capital of France?")
- Current events and news ("latest AI announcements February 2026")
- Finding URLs to investigate further
- Price lookups ("iPhone 16 Pro price")
- Company information ("Anthropic founding year")
- Real-time data that changes frequently
Usage Example
User: "What are the top 3 AI startups that raised Series B in 2026?"
Agent action:
web_search("AI startups Series B funding 2026")
Result: Structured search results with titles, URLs, snippets.
Agent then:
- Summarize the top 3 from search results
- Include source URLs
- Note the date of each source for freshness
Brave Search Parameters
query: Required. The search string.
count: Optional. Number of results (default 10, max 20).
offset: Optional. Pagination offset.
freshness: Optional. Filter by time: "pd" (past day), "pw" (past week), "pm" (past month).
Best Practices for Search Queries
- Be specific: "React 19 new hooks 2026" not "React hooks"
- Include dates for time-sensitive queries: "best laptops February 2026"
- Use quotes for exact phrases:
"Series B" AI startup 2026
- Combine terms for precision:
site:github.com puppeteer screenshot example
- For news, add "news" or "announced" or "latest"
Tier 2 — Web Fetch (web_fetch)
Use web_fetch for:
- Reading a specific URL the user provides
- Following up on a URL found via Brave Search
- Extracting content from documentation pages
- Reading blog posts, articles, or product pages
- Getting raw page content as clean markdown
Usage Example
User: "Read this article and summarize it: https://example.com/blog/ai-trends-2026"
Agent action:
web_fetch("https://example.com/blog/ai-trends-2026")
Result: Cleaned markdown content of the page.
Agent then:
- Provide a structured summary
- Pull out key quotes
- Note the publication date and author
Limitations of Web Fetch
- Returns text content only (no images, no JS execution)
- Will fail on pages that require JavaScript rendering (SPAs)
- Will fail on pages behind authentication
- May return truncated content for very long pages
- Cannot interact with page elements (no clicking, no forms)
When web_fetch fails or returns incomplete content, escalate to Tier 3 (Browser Automation).
Tier 3 — Browser Automation (browser tool)
Use the browser tool when:
- Page requires JavaScript to render content
- You need to interact with elements (click, fill, submit)
- You need a visual screenshot of the page
- You need to navigate through multiple pages in sequence
- Data is loaded dynamically (infinite scroll, AJAX)
- Content is behind a cookie-consent wall or interstitial
Available Browser Actions
| Action | Description |
|---|
navigate | Go to a URL |
screenshot | Capture the current page or a specific element |
click | Click an element by CSS selector |
fill | Type text into an input field by CSS selector |
select | Choose an option from a <select> dropdown |
hover | Hover over an element by CSS selector |
evaluate | Execute arbitrary JavaScript in the page context |
snapshot | Get accessible tree of page elements with refs |
console | Read browser console logs (info, error, warning) |
Browser Patterns
| Pattern | Steps |
|---|
| Navigate + Screenshot | navigate(url) → screenshot(name) → deliver with context |
| Form Fill | navigate → fill(selector, value) → click(submit) → screenshot → evaluate to extract results |
| Table Scraping | navigate → evaluate("Array.from(querySelectorAll('tbody tr')).map(row => ...)") → format as markdown table |
| Multi-page | navigate → evaluate to extract links → navigate each → evaluate + screenshot → compile findings |
| Visual Comparison | navigate(site1) → screenshot → navigate(site2) → screenshot → analyze + compare |
| Login Flow | navigate(login) → fill(email) → ASK user for password → fill(password) → click(submit) → screenshot |
Login rules: NEVER store/log/repeat passwords. ALWAYS ask user for credentials. Screenshot 2FA prompts. Cookies persist within session only.
Browser Profile Selection
When the Chrome Extension Relay is connected, you can choose which browser profile to use:
| Profile | Command | Use When |
|---|
default | browser (no flag) | Public pages, no login needed. Uses headless Chromium on the VM. |
chrome-relay | browser --profile chrome-relay | Login-gated sites (Instagram, Facebook, banking). Uses the user's real Chrome via extension relay. |
Decision flow:
- Check if extension is connected: look for
{"connected":true} from relay status
- If the task needs a logged-in session → use
--profile chrome-relay
- If the task is public browsing → use default (faster, no user browser needed)
- If extension is not connected and login is needed → tell the user to install the extension, then fall back to Tier 3
Tool Decision Matrix
| Query Type | Tool | Why |
|---|
| Factual question ("Who founded OpenAI?") | web_search | Fast, structured answer from search index |
| Current events / news | web_search | Real-time results with freshness filters |
| Read a specific URL | web_fetch | Direct content retrieval, no browser needed |
| Read documentation page | web_fetch | Clean markdown extraction |
| Page requires JS to render | browser | Only browser executes JavaScript |
| Fill out a form | browser | Requires element interaction |
| Take a visual screenshot | browser | Only browser captures rendered pages |
| Scrape a data table | browser | JS evaluation extracts structured data |
| Multi-page workflow | browser | Maintains session across navigations |
| Price comparison across sites | web_search + browser | Search finds URLs, browser extracts prices |
| Site blocked by Cloudflare/WAF (403, CAPTCHA) | crawlee-scrape.py --mode light | TLS fingerprint impersonation bypasses bot detection |
| Blocked even after light Crawlee | crawlee-scrape.py --mode browser | Full Chromium with fingerprint randomization |
| Research workflow (multi-step) | All 3 tiers + Crawlee fallback | Search → Fetch → Browser → Crawlee as needed |
Known Limitations
1. CAPTCHA & Bot Detection
- CAPTCHAs cannot be solved automatically
- If a CAPTCHA appears, screenshot it and inform the user
- Some sites (Cloudflare-protected) may block headless browsers — escalate to
crawlee-scrape.py --mode light which uses TLS impersonation to bypass Cloudflare
- Do NOT attempt to bypass CAPTCHAs manually — this violates platform terms (Crawlee handles Cloudflare challenges automatically)
2. Rate Limits
- Brave Search: 1 query/second, 2000 queries/month (free tier) or 20,000/month (paid)
- Browser sessions: 1 concurrent session per VM
- Rapid-fire page loads may trigger anti-bot protections
- Space requests out by at least 2-3 seconds when crawling multiple pages
3. JavaScript-Heavy SPAs
- Some React/Vue/Angular apps may not fully render on initial load
- Use
evaluate to wait for specific elements before extracting data
- Pattern: Navigate → Wait (evaluate a check) → Extract
- Infinite scroll requires repeated scroll + wait + extract cycles
4. Two-Factor Authentication
- Cannot complete 2FA flows without user providing the code
- Screenshot the 2FA prompt and ask the user for the code
- Time-based OTP codes are time-sensitive — act quickly once provided
5. Anti-Bot Protections
- Headless Chromium (browser tool) may be fingerprinted by advanced bot detection
- Sites using DataDome, PerimeterX, or Akamai Bot Manager may block the browser tool
- Before giving up, escalate to Crawlee:
python3 ~/scripts/crawlee-scrape.py --url "URL" --mode light (try light first, then --mode browser)
- Crawlee uses TLS fingerprint impersonation and browser fingerprint randomization to bypass these protections
- If Crawlee also fails, then inform the user the site has strong anti-bot protection
- Do NOT modify the browser tool's fingerprints directly — use crawlee-scrape.py instead
6. Browser Tool Crashes or Failures
- If
browser → navigate(url) returns "Browser failed" or any error, do NOT go silent.
- First try:
web_fetch(url) — may get partial content without JS.
- Second try:
python3 ~/scripts/crawlee-scrape.py --url "URL" --mode light
- Third try:
python3 ~/scripts/crawlee-scrape.py --url "URL" --mode browser
- If all fail: Tell the user the site blocked automated access and suggest alternatives (e.g., "Can you share a screenshot of the page?" or "I can search for public information about this account instead").
- NEVER go silent after a browser failure. The user is waiting for your response.
Platform Access Status
| Platform | Search | Fetch | Browser | Extension | Notes |
|---|
| Reddit | Works | Works | Works | Works | Old Reddit (old.reddit.com) more reliable for scraping |
| Instagram | Works | Blocked | Blocked | Works | Extension relay uses user's logged-in session. Without extension: use crawlee or ask user for screenshot. |
| Twitter/X | Works | Limited | Limited | Works | Extension relay uses user's logged-in session. Without extension: search results only. |
| Facebook | Works | Blocked | Blocked | Works | Extension relay uses user's logged-in session. Without extension: completely blocked. |
| LinkedIn | Works | Blocked | Limited | Works | Extension relay uses user's logged-in session. Without extension: public profiles only. |
| Amazon | Works | Works | Works | Works | Product pages accessible; may trigger CAPTCHAs on bulk |
| Google | Works (via Brave) | Works | Works | Works | Do not scrape Google directly; use Brave Search API |
| GitHub | Works | Works | Works | Works | Public repos fully accessible; API preferred for data |
| YouTube | Works | Works | Works | Works | Video metadata accessible; transcripts via page scraping |
| eBay | Works | Works | Works | Works | Product listings accessible; watch for pagination |
General rule: If a platform has a public API, prefer the API over scraping. Browser automation is the last resort.
Research Workflow
Escalation ladder: web_search (discover URLs) → web_fetch (read content) → browser (JS rendering/interaction) → crawlee-scrape.py --mode light (anti-bot) → crawlee-scrape.py --mode browser (last resort).
Always: search first, fetch before browser, cite sources with URLs + dates, cross-reference 2+ sources.
Rate Limits & Budget
Brave Search API:
Free tier: 1 req/sec, 2,000 queries/month
Paid tier: 20 req/sec, 20,000 queries/month
Agent budget: 100 searches/day (alert at 80)
Web Fetch:
No hard rate limit
Courtesy: 1 request/second to same domain
Timeout: 30 seconds per request
Browser Sessions:
Concurrent sessions: 1 per VM
Session timeout: 5 minutes of inactivity
Page load timeout: 30 seconds
Max pages per task: 20 (to prevent runaway crawling)
Screenshot storage: /tmp/ (cleared on session end)
Budget Awareness
- Track search count throughout the day
- If approaching daily limit (80+ searches), warn the user
- Prefer
web_fetch over web_search + browser when possible
- Cache results mentally — do not re-search the same query
- For bulk research, plan the search strategy before executing
Dynamic SPA Handling Protocol
SPAs (Instagram, LinkedIn, Facebook, Twitter, banking) load content dynamically. Standard navigate→screenshot fails. Follow these 6 rules:
Rule 1 — Always Wait Before Acting: After every navigate or click, use wait(selector, timeout=10000) before doing anything. Never assume content loaded.
Wait selectors: Instagram DMs [role='listbox'] | Instagram Feed article | LinkedIn .feed-shared-update-v2 | Facebook div[role='feed'] | Twitter article[data-testid='tweet']
Rule 2 — Prefer Snapshots for Data: Use snapshot() (ARIA tree) to extract text and find clickable [ref=N] elements. Use screenshot only for visual verification or showing the user.
Rule 3 — Re-Snapshot After Every Interaction: Element refs go STALE after clicks/scrolls. Always: click(ref) → wait(selector) → snapshot() → use new refs.
Rule 4 — Scroll-to-Load: SPAs use infinite scroll. Content below fold doesn't exist until scrolled. Use evaluate("window.scrollTo(0, document.body.scrollHeight)") → wait(2000) → snapshot(). For containers: evaluate("document.querySelector('[role=\"listbox\"]').scrollTop = container.scrollHeight").
Rule 5 — DOM Queries for Incomplete Snapshots: When snapshot is truncated, use evaluate with querySelectorAll to extract structured data directly. Pattern: Array.from(document.querySelectorAll('SELECTOR')).map(el => ({ text: el.textContent?.trim(), ... })).
Rule 6 — Handle Navigation Failures: If wait times out: check document.readyState, check document.body.innerText.length, take debug screenshot, try alternate selectors, then snapshot() to see what IS on the page.
Instagram DMs Workflow
navigate("instagram.com/direct/inbox/") → wait("[role='listbox']", 15000)snapshot() → find conversation refsclick(ref=N) → wait("[role='row']", 10000) → snapshot()- Scroll to load older:
evaluate("querySelector('[role=\"listbox\"]').scrollTop = 0") → wait(3000) → snapshot()
- Extract:
evaluate("querySelectorAll('[role=\"row\"]').map(el => el.textContent?.trim()?.substring(0,500))")
- Send:
snapshot() → fill(ref=N, value) → press("Enter") → wait(2000) → snapshot()
Key Rules
- Use
web_search first (10x faster than browser). Never scrape Google directly — use Brave API.
- Try
web_fetch before browser. If fetch fails (JS-rendered page), escalate to browser.
- If browser gets 403/CAPTCHA/Cloudflare: try
crawlee-scrape.py --mode light, then --mode browser.
- Always cite sources with URL + retrieval date.
- Always wait for SPA elements before extracting —
evaluate with polling or wait(selector).
- Cross-reference at least 2 sources for important facts.
