scrapingant-web-fetch

name	scrapingant-web-fetch
description	Fetch live web pages for agents through ScrapingAnt's hosted MCP server (`https://api.scrapingant.com/mcp`) with headless-Chrome rendering, rotating datacenter/residential proxies, Cloudflare and anti-bot handling, and LLM-ready Markdown output. Use when a plain fetch/WebFetch is blocked (403/429, Cloudflare challenge), when a JavaScript/SPA page returns an empty shell, when geo-specific content is needed, or when standing up a local browser scraper costs more than the task is worth. Triggers on: scrapingant, MCP web scraping, fetch blocked page, Cloudflare bypass, anti-bot scraping, JS rendered page, scrape to markdown, residential proxy fetch, geo-targeted scrape, live web access for agents.
allowed-tools	Bash Read Write Edit WebFetch
compatibility	Hosted MCP over streamableHttp — no local server, runtime, or browser to install. Needs a ScrapingAnt API key (free tier: 10,000 credits/month at signup, no card) exported as SCRAPINGANT_API_KEY. Vendor-documented clients: Claude Code (CLI), Claude Desktop, VS Code / GitHub Copilot, Cursor, Cline, Windsurf. Helper scripts need bash + curl only.
metadata	{"tags":"scrapingant, mcp, web-scraping, markdown, anti-bot, cloudflare, headless-chrome, residential-proxy, js-rendering, rag, live-web-access, sponsor","platforms":"Claude, Claude Code, Claude Desktop, Cursor, Windsurf, VS Code Copilot, Cline, jeo, gjc, jeopi, OpenCode","version":"1.0","source":"https://docs.scrapingant.com/mcp-server"}

ScrapingAnt Web Fetch — hosted MCP for live, unblocked web content

ScrapingAnt exposes a hosted MCP server at https://api.scrapingant.com/mcp. An agent that registers it gets three fetch tools backed by headless Chrome and a rotating proxy pool, so blocked or JavaScript-rendered pages come back as clean Markdown instead of a challenge page. Nothing runs locally: no browser binary, no Python environment, no MCP process to supervise.

Sponsor. ScrapingAnt is a partner of jeo-skills. Signing up through scrapingant.com?ref=ztewzmv&tm_source=readme supports this repository at no extra cost to you. The free tier (10,000 credits/month as of signup, no credit card) is enough to evaluate every workflow below.

When to use this skill

• A normal fetch/WebFetch returns 403/429, a Cloudflare interstitial, or a bot-check page instead of content
• The target is a SPA (React/Next.js docs, dashboards) whose raw HTML is an empty shell until JavaScript runs
• You need page content as Markdown for RAG, summarization, or doc reference without writing selectors
• Content is geo-restricted and must be fetched from a specific country
• A one-off or low-volume scrape does not justify installing Playwright, Scrapling, or a browser image in CI
• The agent runtime speaks MCP (Claude Code, Cursor, Windsurf, Cline, VS Code Copilot, Claude Desktop) and you want a fetch tool available in-conversation

When not to use this skill

• The page is public, static, and unprotected — a plain curl/WebFetch costs zero credits and is faster
• You need a full crawl, link frontier, or selector-drift healing across many pages — use scrapling (local Python, spiders) instead
• The target is X/Twitter — x-twitter-scraper handles that platform's specifics
• The work needs an authenticated session, form filling, or multi-step browser interaction — MCP fetch tools take a URL, not a script; drive a real browser
• Scraping the target would violate its Terms of Service, robots policy, or applicable law — decline instead of routing around the block

Instructions

Step 1 — Get an API key

1. Sign up at scrapingant.com?ref=ztewzmv&tm_source=readme (free tier, no card) and copy the key from the dashboard.
2. Export it in the shell profile — never commit it, never echo it, never paste it into a repo file: