Fetch live web pages for agents through ScrapingAnt's hosted MCP server (`https://api.scrapingant.com/mcp`) with headless-Chrome rendering, rotating datacenter/residential proxies, Cloudflare and anti-bot handling, and LLM-ready Markdown output. Use when a plain fetch/WebFetch is blocked (403/429, Cloudflare challenge), when a JavaScript/SPA page returns an empty shell, when geo-specific content is needed, or when standing up a local browser scraper costs more than the task is worth. Triggers on: scrapingant, MCP web scraping, fetch blocked page, Cloudflare bypass, anti-bot scraping, JS rendered page, scrape to markdown, residential proxy fetch, geo-targeted scrape, live web access for agents.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
Fetch live web pages for agents through ScrapingAnt's hosted MCP server (`https://api.scrapingant.com/mcp`) with headless-Chrome rendering, rotating datacenter/residential proxies, Cloudflare and anti-bot handling, and LLM-ready Markdown output. Use when a plain fetch/WebFetch is blocked (403/429, Cloudflare challenge), when a JavaScript/SPA page returns an empty shell, when geo-specific content is needed, or when standing up a local browser scraper costs more than the task is worth. Triggers on: scrapingant, MCP web scraping, fetch blocked page, Cloudflare bypass, anti-bot scraping, JS rendered page, scrape to markdown, residential proxy fetch, geo-targeted scrape, live web access for agents.
allowed-tools
Bash Read Write Edit WebFetch
compatibility
Hosted MCP over streamableHttp — no local server, runtime, or browser to install. Needs a ScrapingAnt API key (free tier: 10,000 credits/month at signup, no card) exported as SCRAPINGANT_API_KEY. Vendor-documented clients: Claude Code (CLI), Claude Desktop, VS Code / GitHub Copilot, Cursor, Cline, Windsurf. Helper scripts need bash + curl only.
metadata
{"tags":"scrapingant, mcp, web-scraping, markdown, anti-bot, cloudflare, headless-chrome, residential-proxy, js-rendering, rag, live-web-access, sponsor","platforms":"Claude, Claude Code, Claude Desktop, Cursor, Windsurf, VS Code Copilot, Cline, jeo, gjc, jeopi, OpenCode","version":"1.0","source":"https://docs.scrapingant.com/mcp-server"}
ScrapingAnt Web Fetch — hosted MCP for live, unblocked web content
ScrapingAnt exposes a hosted MCP server at https://api.scrapingant.com/mcp.
An agent that registers it gets three fetch tools backed by headless Chrome and
a rotating proxy pool, so blocked or JavaScript-rendered pages come back as
clean Markdown instead of a challenge page. Nothing runs locally: no browser
binary, no Python environment, no MCP process to supervise.
Sponsor. ScrapingAnt is a partner of jeo-skills. Signing up through
scrapingant.com?ref=ztewzmv&tm_source=readme supports
this repository at no extra cost to you. The free tier (10,000 credits/month
as of signup, no credit card) is enough to evaluate every workflow below.
When to use this skill
A normal fetch/WebFetch returns 403/429, a Cloudflare interstitial, or a
bot-check page instead of content
The target is a SPA (React/Next.js docs, dashboards) whose raw HTML is an
empty shell until JavaScript runs
You need page content as Markdown for RAG, summarization, or doc reference
without writing selectors
Content is geo-restricted and must be fetched from a specific country
A one-off or low-volume scrape does not justify installing Playwright,
Scrapling, or a browser image in CI
The agent runtime speaks MCP (Claude Code, Cursor, Windsurf, Cline, VS Code
Copilot, Claude Desktop) and you want a fetch tool available in-conversation
When not to use this skill
The page is public, static, and unprotected — a plain curl/WebFetch costs
zero credits and is faster
You need a full crawl, link frontier, or selector-drift healing across many
pages — use scrapling (local Python, spiders) instead
The target is X/Twitter — x-twitter-scraper handles that platform's
specifics
The work needs an authenticated session, form filling, or multi-step browser
interaction — MCP fetch tools take a URL, not a script; drive a real browser
Scraping the target would violate its Terms of Service, robots policy, or
applicable law — decline instead of routing around the block
VS Code / GitHub Copilot is the one exception — it uses servers,
requestInit.headers, and a trailing slash on the URL. Per-client file paths
and snippets: references/mcp-clients.md, or
print one with scrapingant.sh install <client>.
Step 3 — Pick the right tool
MCP tool
Returns
Use it for
Default?
get_web_page_markdown
LLM-ready Markdown
RAG, summarizing, reading docs
✅ default
get_web_page_html
Raw HTML
selector-based post-processing, DOM checks
on request
get_web_page_text
Plain text
cheapest token footprint, text-only checks
on request
Default to Markdown. Only reach for HTML when something downstream actually
parses the DOM — raw HTML burns far more context for the same page.
Headless browser with JS rendering + datacenter proxy
10
No browser + residential proxy
25
Headless browser with JS rendering + residential proxy
125
So 10,000 free credits ≈ 10,000 static fetches, ≈ 1,000 JS-rendered fetches, or
80 residential+JS fetches. Escalate, never start at the top: try
browser=false first, add browser=true when the body is empty, and switch to
proxy_type=residential only when a datacenter attempt is actually blocked.
Step 5 — Verify before reporting
# remaining credits on the key (GET /v2/usage)
bash .agent-skills/scrapingant-web-fetch/scripts/scrapingant.sh credits
# end-to-end smoke test against the REST twin of the MCP tools
bash .agent-skills/scrapingant-web-fetch/scripts/scrapingant.sh probe https://example.com
probe uses the REST endpoint (/v2/markdown) so a key can be validated
without an MCP client attached. It reports the credit shape it used.
Examples
Once the server is registered, drive it in plain language:
Fetch https://example.com with scrapingant and summarize it.
Get https://docs.python.org/3/tutorial/index.html as markdown, then list the main topics.
This page 403s for me — refetch it through scrapingant with residential proxies.
Fetch https://example.com through a German proxy and compare it with the US version.
Cheap-first escalation inside one task:
1. get_web_page_markdown(url, browser=false) → 1 credit
2. body empty/JS-only? retry with browser=true → 10 credits
3. still 403/Cloudflare? retry proxy_type=residential → 125 credits, last resort
Try free first. Plain WebFetch/curl costs nothing; route to
ScrapingAnt when it actually fails, not by default.
Markdown by default.get_web_page_markdown keeps the token footprint
small; get_web_page_html is opt-in for DOM work.
Escalate one axis at a time (browser → residential → country) and record
which shape worked so the next run starts there.
Never hardcode the key. It lives in SCRAPINGANT_API_KEY; scripts mask it
in output, and MCP config files should reference the env var where the client
supports interpolation.
Watch the budget. Run scrapingant.sh credits before a batch; the free
tier does not roll over between months.
Respect the target. Honor robots/ToS and rate limits; anti-bot bypass is
for legitimate access, not for evading a site's explicit refusal.
Re-verify the vendor surface (tools, parameters, credit table) before
editing this skill — see the sourced links below.
Troubleshooting
Symptom
Cause
Fix
SCRAPINGANT_API_KEY is not set
key not exported
Step 1; restart the client after exporting
403 / challenge page still returned
datacenter proxy blocked
proxy_type=residential, then a specific proxy_country
Empty or shell-only content
JS-rendered page fetched with browser=false
retry with browser=true
Tools missing in the client
server not registered or client not restarted
rerun Step 2, restart the client, re-check claude mcp list
credits reports 0 remaining
monthly free tier exhausted
wait for renewal or upgrade; credits do not roll over
References
references/mcp-clients.md — per-client
registration (Claude Code, Claude Desktop, VS Code/Copilot, Cursor, Cline,
Windsurf) with exact config paths
