web-scraping

name	web-scraping
description	Crawl and extract content from websites. Saves pages as Markdown, HTML, or text into the web-crawling/ folder, organized by site. Supports static pages, JavaScript-heavy sites, and sites with bot protection.

Web Scraping

This skill lets you crawl websites and extract their content. All results are saved to web-crawling/ inside the project directory, organized by site name. The skill connects to the web-scraper MCP server which provides six tools at different protection levels.

Dependencies

This skill requires the web-scraper service to be running (port 3480). If it is not running, start it via the process manager before proceeding.

When to Use This Skill

Activate this skill when the user wants to:

• Scrape, crawl, or download content from one or more web pages
• Extract specific parts of a website using CSS selectors
• Save website content for offline reading or analysis
• Collect data from multiple URLs in bulk
• Access pages that require JavaScript rendering
• Access pages behind Cloudflare or other bot protections

Trigger phrases include: "scrape this page", "crawl this website", "grab the content from", "download this page", "extract data from", "fetch this URL".

What You Can Do

When the user asks about web scraping capabilities, explain clearly in plain language:

Three levels of page retrieval, from fast to thorough:

1. Quick retrieval — Fast HTTP requests that work on most regular websites (blogs, news sites, documentation, wikis). This is the default and fastest option. It mimics a real browser's network signature to avoid basic blocks.

2. Browser-based retrieval — Opens a real headless browser to load the page. Use this when the website relies on JavaScript to render its content (single-page apps, dynamic dashboards, infinite scroll pages). Slower than quick retrieval but sees the page as a real user would.

3. Stealth retrieval — A hardened browser mode designed to bypass advanced bot detection (Cloudflare challenges, CAPTCHAs, aggressive fingerprinting). Use this as a last resort when the other methods get blocked. Can also solve Cloudflare Turnstile challenges automatically.

Content extraction options:

• Markdown (default) — Clean, readable text with formatting preserved. Best for most use cases.
• HTML — Raw HTML source. Useful when the user needs the exact page structure.
• Text — Plain text with no formatting. Useful for data processing.

CSS selector filtering — The user can request only specific parts of a page (e.g., "just the article body", "only the product prices", "the main table"). You can pass CSS selectors to narrow down what gets returned, dramatically reducing noise.

Bulk operations — All three retrieval levels support fetching multiple URLs at once in parallel, which is much faster than fetching them one by one.

Limitations

Be upfront with the user about these limitations:

• No login/authentication — Cannot access pages behind a login form. The tools can send custom cookies and headers, but cannot interactively log in to a website.
• No recursive crawling — Each request fetches one page (or a list of explicit URLs). It does not automatically follow links to discover and crawl an entire site. You must identify and provide the URLs yourself.
• Rate limits and blocking — Websites may block or rate-limit requests. Space out requests when crawling multiple pages from the same domain. There is no built-in delay between requests, so add pauses yourself when needed.
• Large pages — Very large pages may produce large output. Use CSS selectors to limit what gets extracted.
• Dynamic content timing — For browser-based retrieval, some pages may need extra wait time for JavaScript to finish loading. You can configure timeouts and wait conditions, but some very slow pages may still return incomplete content.

File & Folder Conventions

Output directory

All scraped content goes into web-crawling/ at the project root. Create it if it does not exist.

Site folders

Inside web-crawling/, create a subfolder for each website, named after the domain (lowercase, hyphens for dots):

• https://example.com/page → web-crawling/example-com/
• https://docs.python.org/3/ → web-crawling/docs-python-org/
• https://blog.example.co.uk/post → web-crawling/blog-example-co-uk/

File naming

Name files after the page path, sanitized to be filesystem-safe. Use the extraction type as extension:

• web-crawling/example-com/about.md
• web-crawling/example-com/products/widget-123.md
• web-crawling/docs-python-org/3/library/json.md

For the root page (/), use index.md.

Index file

Maintain web-crawling/crawl-log.md as a log of all crawled pages:

# Web Crawling Log

_Last updated: <date>_

## example-com

| Page | File | Retrieved | Method |
|------|------|-----------|--------|
| https://example.com/ | [index.md](web-crawling/example-com/index.md) | 2026-02-27 | quick |
| https://example.com/about | [about.md](web-crawling/example-com/about.md) | 2026-02-27 | quick |

## docs-python-org

| Page | File | Retrieved | Method |
|------|------|-----------|--------|
| https://docs.python.org/3/library/json.html | [3/library/json.md](web-crawling/docs-python-org/3/library/json.md) | 2026-02-27 | browser |

Update this log after every crawl operation.

Workflow

1. Understand the request

Ask the user what they want to scrape:

• Which URL(s)?
• Do they need the full page or specific parts (CSS selector)?
• What format (markdown, HTML, text)? Default to markdown if not specified.

2. Choose the retrieval method

Select the appropriate tool based on the target site:

Situation	Tool	Why
Regular static website	get	Fast, efficient, works for most sites
Multiple URLs from the same site	bulk_get	Parallel fetching, much faster
JavaScript-rendered page (SPA, React, etc.)	fetch	Needs a real browser to render
Multiple JS-rendered pages	bulk_fetch	Parallel browser tabs
Site with bot protection (Cloudflare, etc.)	stealthy_fetch	Bypasses advanced detection
Multiple protected pages	bulk_stealthy_fetch	Parallel stealth sessions

Start with get by default. Only escalate to fetch or stealthy_fetch if:

• The user says the site uses JavaScript rendering
• The get tool returns empty or minimal content (likely JS-rendered)
• The get or fetch tool gets blocked (403, CAPTCHA page, Cloudflare challenge)

When escalating, inform the user: "The page appears to need browser rendering, switching to a more thorough retrieval method."

3. Make the request

Call the appropriate MCP tool. Key parameters to use:

For get / bulk_get:

• url / urls: The target URL(s)
• extraction_type: "markdown" (default), "html", or "text"
• css_selector: Optional CSS selector to narrow results
• main_content_only: true (default) — extracts only the body content, skipping navbars, footers, etc.

For fetch / bulk_fetch:

• Same as above, plus:
• network_idle: Set to true for pages that load content dynamically after initial page load
• wait: Milliseconds to wait after page load (useful for slow-loading content)
• wait_selector: CSS selector to wait for before extracting (e.g., wait until a specific element appears)
• disable_resources: Set to true to skip loading images/fonts/media for faster page loads

For stealthy_fetch / bulk_stealthy_fetch:

• Same as fetch, plus:
• solve_cloudflare: Set to true to automatically solve Cloudflare Turnstile challenges

4. Save the results

1. Create the directory structure: web-crawling/<site-folder>/
2. Write the extracted content to the appropriate file
3. Update web-crawling/crawl-log.md

5. Present results

After saving:

• Confirm what was saved and where
• Show a brief preview of the content (first few lines)
• If the content seems incomplete or was blocked, suggest trying a different retrieval method

MCP Tools Reference

All tools are provided by the web-scraper MCP server.

get

Fast HTTP request with browser fingerprint impersonation. Best for static websites.

• url (string, required): The URL to fetch
• extraction_type (string): "markdown", "html", or "text". Default: "markdown"
• css_selector (string): CSS selector to extract specific elements
• main_content_only (bool): Extract only body content. Default: true
• Returns: Structured response with content, url, status_code

bulk_get

Fetch multiple URLs in parallel via HTTP.

• urls (list of strings, required): URLs to fetch
• Same options as get
• Returns: List of responses, one per URL

fetch

Browser-based fetching using headless Chromium. For JavaScript-rendered pages.

• url (string, required): The URL to fetch
• extraction_type, css_selector, main_content_only: Same as get
• network_idle (bool): Wait for no network activity. Default: false
• wait (number): Extra milliseconds to wait after load. Default: 0
• wait_selector (string): CSS selector to wait for before extracting
• disable_resources (bool): Skip images/fonts/media. Default: false
• timeout (number): Timeout in milliseconds. Default: 30000

bulk_fetch

Fetch multiple URLs in parallel browser tabs.

• urls (list of strings, required): URLs to fetch
• Same options as fetch
• Returns: List of responses

stealthy_fetch

Hardened browser mode for sites with advanced bot protection.

• Same options as fetch, plus:
• solve_cloudflare (bool): Auto-solve Cloudflare challenges. Default: false
• block_webrtc (bool): Prevent local IP leak. Default: false
• hide_canvas (bool): Prevent canvas fingerprinting. Default: false

bulk_stealthy_fetch

Parallel stealth fetching for multiple protected URLs.

• urls (list of strings, required): URLs to fetch
• Same options as stealthy_fetch
• Returns: List of responses

Example Interactions

Example 1: Simple page scrape

User: "Scrape the Python json module documentation at https://docs.python.org/3/library/json.html"

Agent:

1. Calls get with url https://docs.python.org/3/library/json.html, extraction_type "markdown"
2. Creates web-crawling/docs-python-org/3/library/json.md
3. Updates crawl log

"I've saved the Python json module documentation to web-crawling/docs-python-org/3/library/json.md. Here's a preview:

json - JSON encoder and decoder

Source code: Lib/json/init.py JSON (JavaScript Object Notation) is a lightweight data interchange format..."

Example 2: Multiple pages with CSS selector

User: "Grab just the article content from these three blog posts: [url1], [url2], [url3]"

Agent:

1. Calls bulk_get with the three URLs, css_selector "article" or ".post-content" (infer from context or try common selectors)
2. Saves each to web-crawling/<domain>/...
3. Updates crawl log

"Done! I've saved all three articles to the web-crawling/ folder. I used a CSS selector to grab just the article body, skipping navigation and sidebars."

Example 3: JavaScript-rendered page (escalation)

User: "Scrape this React dashboard at https://app.example.com/dashboard"

Agent:

1. First tries get — content comes back nearly empty
2. Informs user: "The page appears to use JavaScript rendering. Switching to browser-based retrieval."
3. Calls fetch with url, network_idle true
4. Saves result

Example 4: Cloudflare-protected site

User: "I need the content from https://protected-site.com but it has Cloudflare"

Agent:

1. Calls stealthy_fetch with solve_cloudflare true
2. Saves result

"I've retrieved the page using stealth mode with Cloudflare challenge solving. Content saved to web-crawling/protected-site-com/index.md."

Error Handling

• Empty or minimal content from get: Suggest escalating to fetch (likely JS-rendered)
• 403 / blocked response: Suggest escalating to stealthy_fetch
• Timeout: Suggest increasing the timeout parameter or using wait for slow pages
• CSS selector returns nothing: Inform the user the selector didn't match. Suggest trying without the selector first to see the full page structure, then refining.
• Web-scraper service not running: Inform the user that the Web Scraper service needs to be started via the process manager.

Notes

• Always default to extraction_type: "markdown" unless the user specifically asks for HTML or text. Markdown is the most useful format for further analysis and is the most token-efficient.
• When scraping multiple pages from the same domain, consider adding a brief pause between requests to be respectful of the server.
• The main_content_only parameter (default true) strips navigation, footers, and sidebars. Disable it only if the user explicitly needs the full page structure.
• Domain folder names are derived by replacing dots with hyphens: docs.python.org becomes docs-python-org.