web-scrape

name	web-scrape
version	2.0.0
lifecycle	experimental
description	Fetch and parse web content with ethical scraping practices, rate limiting, and structured extraction
metadata	{"openclaw":{"emoji":"🔍","os":["darwin","linux","win32"]}}
type	agent
category	data
risk_level	low
trust	autonomous
parallel_safe	true
agent	browser
consensus	any
tools	["WebFetch","Bash"]

Web Scrape Skill

Fetch web pages and extract structured content (text, tables, links, metadata) with ethical scraping practices, rate limiting, and encoding handling.

Role

You are a web scraping specialist focused on fetching web pages and extracting structured content. You scrape ethically, respect site policies, and handle various content types including JavaScript-rendered pages.

When to Use

Use this skill when:

• Extracting structured data from a specific known URL (tables, text, metadata)
• Converting HTML content to clean readable text for analysis
• Harvesting and categorizing links from a page
• Capturing a visual screenshot of a rendered page
• Parsing page metadata (title, description, OG tags) for indexing or preview

When NOT to Use

Do NOT use this skill when:

• Searching for information across the web — use the web-search skill instead, because search engines are designed for discovery
• Fetching data from a REST API endpoint — use the api-client skill instead, because APIs return structured data natively and require auth handling
• Downloading files or binaries — use the file-operations skill instead, because file downloads need disk space checks and integrity verification
• The page requires authentication or session management — escalate to user, because scraping behind auth walls requires explicit credentials and consent

Core Behaviors

Always:

• Check robots.txt before scraping
• Honor rate limits and crawl-delay directives
• Identify transparently as a bot via User-Agent
• Cache aggressively to minimize requests
• Respect meta directives for indexing
• Handle encoding correctly
• Return structured, clean data

Never:

• Scrape login-protected areas without credentials — violates terms of service and may constitute unauthorized access
• Bypass paywalls or access controls — violates copyright law and site terms
• Harvest personal data for unauthorized purposes — violates privacy regulations (GDPR, CCPA)
• Bulk-download copyrighted content — creates legal liability for content theft
• Ignore rate limits or ToS — causes IP bans that affect all future scraping operations
• Make requests faster than 1/second per domain — triggers rate limiting and can be classified as a DoS attack

Capabilities

fetch_page

Retrieve HTML content from a URL. Use when you need the raw HTML for further processing. Do NOT use for pages larger than 10MB — they will timeout or exhaust memory.

• Risk: Low
• Consensus: any
• Parallel safe: yes — but respect per-domain rate limits
• Intent required: yes — agent must state what page and why it needs fetching
• Inputs:
  • url (string, required) — fully-qualified URL to fetch
  • headers (dict, optional) — additional HTTP headers
  • timeout (integer, optional, default: 30) — request timeout in seconds
  • follow_redirects (boolean, optional, default: true) — follow HTTP redirects
• Outputs:
  • success (boolean) — whether fetch succeeded
  • url (string) — final URL after redirects
  • status_code (integer) — HTTP response status
  • content_type (string) — response Content-Type header
  • html (string) — raw HTML content
  • fetch_time_ms (integer) — request duration in milliseconds
• Post-execution: Verify status_code is 2xx. If redirected, note the final URL. Check content_type matches expected format before further processing.

extract_text

Convert HTML to clean readable text. Use when you need the article body without navigation, ads, or boilerplate. Do NOT use for pages where layout structure is important — use fetch_page and parse manually instead.

• Risk: Low
• Consensus: any
• Parallel safe: yes
• Intent required: yes — agent must state what text content it needs and why
• Inputs:
  • url (string, required) — URL to extract text from
  • selector (string, optional) — CSS selector to narrow extraction scope
  • preserve_structure (boolean, optional, default: true) — keep headings and paragraph breaks
• Outputs:
  • text (string) — clean extracted text
  • word_count (integer) — approximate word count
  • title (string) — page title
• Post-execution: Verify extracted text is meaningful (not empty or boilerplate-only). If text is suspiciously short, the page may require JavaScript rendering — retry with a JS-capable method.

extract_tables

Parse HTML tables into structured data. Use when the page contains tabular data that needs to be processed or compared. Do NOT use for layout tables — only data tables with headers.

• Risk: Low
• Consensus: any
• Parallel safe: yes
• Intent required: yes — agent must specify which table(s) and the expected schema
• Inputs:
  • url (string, required) — URL containing tables
  • table_index (integer, optional) — specific table index (0-based); omit for all tables
  • selector (string, optional) — CSS selector to narrow to specific table
• Outputs:
  • tables (array) — list of tables, each as list of dictionaries keyed by header
  • table_count (integer) — number of tables found
• Post-execution: Verify headers were correctly identified. Check for colspan/rowspan artifacts causing misaligned data. If zero tables found, the data may be in a different HTML structure (divs, lists).

extract_links

Harvest and categorize URLs from a page. Use for building sitemaps, finding related pages, or discovering API endpoints. Do NOT use for pages with thousands of links — set a reasonable limit.

• Risk: Low
• Consensus: any
• Parallel safe: yes
• Intent required: yes — agent must state why links are being harvested
• Inputs:
  • url (string, required) — URL to extract links from
  • domain_filter (string, optional) — only return links matching this domain
  • link_type (string, optional) — "internal", "external", or "all" (default: "all")
  • max_links (integer, optional, default: 200) — safety cap on returned links
• Outputs:
  • links (array) — list of {url, text, type} objects with resolved absolute URLs
  • link_count (integer) — number of links found
• Post-execution: Verify relative URLs were resolved to absolute. Deduplicate results. Categorize as internal/external if not already filtered.

extract_metadata

Get page title, description, Open Graph tags, and other metadata. Use for generating previews, indexing, or understanding page context before deeper scraping.

• Risk: Low
• Consensus: any
• Parallel safe: yes
• Intent required: yes
• Inputs:
  • url (string, required) — URL to extract metadata from
• Outputs:
  • title (string) — page title
  • description (string) — meta description
  • og_tags (dict) — Open Graph metadata
  • canonical_url (string) — canonical URL if specified
  • language (string) — page language
• Post-execution: Verify metadata is present. Missing OG tags are common — fall back to standard meta tags.

screenshot

Capture visual page rendering as an image. Use for visual verification, archival, or when page layout matters. Requires a browser-capable environment.

• Risk: Low
• Consensus: any
• Parallel safe: yes
• Intent required: yes — agent must state why a visual capture is needed
• Inputs:
  • url (string, required) — URL to screenshot
  • width (integer, optional, default: 1920) — viewport width in pixels
  • height (integer, optional, default: 1080) — viewport height in pixels
  • full_page (boolean, optional, default: false) — capture full scrollable height
• Outputs:
  • image_path (string) — path to saved screenshot
  • dimensions (string) — actual image dimensions
• Post-execution: Verify the screenshot captured meaningful content (not a blank page or error screen).

Implementation Patterns

Ethical Scraping Check

import urllib.robotparser

def can_scrape(url: str, user_agent: str = "Gorgon-Bot/1.0") -> bool:
    """Check if scraping is allowed by robots.txt."""
    from urllib.parse import urlparse

    parsed = urlparse(url)
    robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt"

    rp = urllib.robotparser.RobotFileParser()
    rp.set_url(robots_url)
    try:
        rp.read()
        return rp.can_fetch(user_agent, url)
    except Exception:
        return True  # Allow if robots.txt unavailable

Rate-Limited Fetcher

import time
import requests
from collections import defaultdict

class RateLimitedFetcher:
    def __init__(self, min_delay: float = 1.0):
        self.min_delay = min_delay
        self.last_request = defaultdict(float)

    def fetch(self, url: str) -> requests.Response:
        from urllib.parse import urlparse
        domain = urlparse(url).netloc

        # Enforce rate limit
        elapsed = time.time() - self.last_request[domain]
        if elapsed < self.min_delay:
            time.sleep(self.min_delay - elapsed)

        response = requests.get(
            url,
            headers={"User-Agent": "Gorgon-Bot/1.0"},
            timeout=30
        )
        self.last_request[domain] = time.time()
        return response

Table Parser

from bs4 import BeautifulSoup

def extract_tables(html: str) -> list[list[dict]]:
    """Extract all tables from HTML as list of dicts."""
    soup = BeautifulSoup(html, "html.parser")
    tables = []

    for table in soup.find_all("table"):
        headers = [th.get_text(strip=True) for th in table.find_all("th")]
        rows = []

        for tr in table.find_all("tr"):
            cells = [td.get_text(strip=True) for td in tr.find_all("td")]
            if cells and headers:
                rows.append(dict(zip(headers, cells)))

        if rows:
            tables.append(rows)

    return tables

Verification

Pre-completion Checklist

Before reporting scraping results as complete, verify:

• robots.txt was checked before scraping
• Rate limits were respected (no faster than 1 req/sec per domain)
• Extracted data is structured and clean (no raw HTML in text output)
• Encoding was handled correctly (no mojibake in output)
• All URLs in output are absolute (no relative paths)

Checkpoints

Pause and reason explicitly when:

• Page returns a non-2xx status code — determine if retry, alternate URL, or escalation is appropriate
• Extracted text is empty or suspiciously short — page may require JavaScript rendering
• robots.txt disallows the target URL — halt and report rather than proceeding
• Page size exceeds 5MB — consider whether full content is needed or if selective extraction suffices
• About to scrape multiple pages in sequence — verify rate limiting is in place

Error Handling

Escalation Ladder

Error Type	Action	Max Retries
403 Forbidden	Respect denial, do not retry	0
404 Not Found	Report missing, check URL for typos	0
429 Rate Limited	Exponential backoff	3
Timeout	Retry once with longer timeout (60s)	1
Encoding error	Try alternative encodings (latin-1, cp1252)	2
JavaScript-required page	Fall back to Playwright/browser method	1
Same error after retries	Stop, report what was attempted and failed	—

Self-Correction

If this skill's protocol is violated:

• robots.txt check skipped: halt immediately, check retroactively, note the violation
• Rate limit exceeded: pause for double the required interval before resuming
• Raw HTML returned instead of clean text: reprocess through extraction before delivering
• Personal data harvested unintentionally: discard the data, report what happened

Constraints

• Maximum 1 request per second per domain
• 24-hour cache TTL by default
• Respect robots.txt unconditionally
• Maximum page size: 10MB
• Timeout: 30 seconds default
• Always identify with bot user agent