| name | fetch-with-browser-fallback |
| description | Fetch a URL that blocks web_fetch by falling back to the browser tool |
| tags | ["web","fallback","browser"] |
| tools_required | ["browser"] |
When to use
When web_fetch returns 403 Forbidden (bot detection) on a URL.
Steps
- Call
browser(action="start") to launch the browser session
- Call
browser(action="navigate", url="<target_url>") to load the page
- Call
browser(action="snapshot") to extract the page content as an accessibility tree
- Call
browser(action="stop") to close the browser session
- Parse the article content from the snapshot (look for heading + paragraph text)
When web_fetch returns anti-bot or empty cache
If web_fetch returns a 200 status but the content is an anti-bot redirect or empty cache:
- Check the content for anti-bot phrases (e.g., "Please click here if you are not redirected within a few seconds")
- Try the Wayback Machine API:
https://archive.org/wayback/available?url=<target_url>
- If no archive, search for the site's documentation or API docs
- Use the browser to navigate to the documentation and find the specific page
Known blocked domains (browser also fails)
- cnbc.com
- goodrx.com — aggressive Cloudflare/WAF, blocks both
web_fetch and headless Chromium. Use web_search index instead.
- Likely: wsj.com, bloomberg.com, reuters.com, nytimes.com (news sites with anti-bot measures)
Notes
- The browser tool uses a real Chromium session with proper headers, so it bypasses most bot detection
- Do NOT retry
web_fetch — it will fail again on the same domain
- If the browser also fails, the content may be behind a paywall requiring login, or the site has aggressive anti-bot (WAF). Fall back to
web_search for indexed content.
- Google Cache may return anti-bot redirects even with a 200 status — check content before assuming success
