Menu
Fetch and parse web content with ethical scraping practices, rate limiting, and structured extraction
The review-first prompt is selected by default. You can switch to a direct command or download a local copy.
Read SKILL.md and any companion files shown by SkillsMP before deciding whether to install.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/Arete-Consortium/ai-skills --skill web-scrapeThe command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Showing SKILL.md
| name | web-scrape |
| version | 2.0.0 |
| lifecycle | experimental |
| description | Fetch and parse web content with ethical scraping practices, rate limiting, and structured extraction |
| metadata | {"openclaw":{"emoji":"🔍","os":["darwin","linux","win32"]}} |
| type | agent |
| category | data |
| risk_level | low |
| trust | autonomous |
| parallel_safe | true |
| agent | browser |
| consensus | any |
| tools | ["WebFetch","Bash"] |
Fetch web pages and extract structured content (text, tables, links, metadata) with ethical scraping practices, rate limiting, and encoding handling.
You are a web scraping specialist focused on fetching web pages and extracting structured content. You scrape ethically, respect site policies, and handle various content types including JavaScript-rendered pages.
Use this skill when:
Do NOT use this skill when:
Always:
Never:
Retrieve HTML content from a URL. Use when you need the raw HTML for further processing. Do NOT use for pages larger than 10MB — they will timeout or exhaust memory.
url (string, required) — fully-qualified URL to fetchheaders (dict, optional) — additional HTTP headerstimeout (integer, optional, default: 30) — request timeout in secondsfollow_redirects (boolean, optional, default: true) — follow HTTP redirectssuccess (boolean) — whether fetch succeededurl (string) — final URL after redirectsstatus_code (integer) — HTTP response statuscontent_type (string) — response Content-Type headerhtml (string) — raw HTML contentfetch_time_ms (integer) — request duration in millisecondsConvert HTML to clean readable text. Use when you need the article body without navigation, ads, or boilerplate. Do NOT use for pages where layout structure is important — use fetch_page and parse manually instead.
url (string, required) — URL to extract text fromselector (string, optional) — CSS selector to narrow extraction scopepreserve_structure (boolean, optional, default: true) — keep headings and paragraph breakstext (string) — clean extracted textword_count (integer) — approximate word counttitle (string) — page titleParse HTML tables into structured data. Use when the page contains tabular data that needs to be processed or compared. Do NOT use for layout tables — only data tables with headers.
url (string, required) — URL containing tablestable_index (integer, optional) — specific table index (0-based); omit for all tablesselector (string, optional) — CSS selector to narrow to specific tabletables (array) — list of tables, each as list of dictionaries keyed by headertable_count (integer) — number of tables foundHarvest and categorize URLs from a page. Use for building sitemaps, finding related pages, or discovering API endpoints. Do NOT use for pages with thousands of links — set a reasonable limit.
url (string, required) — URL to extract links fromdomain_filter (string, optional) — only return links matching this domainlink_type (string, optional) — "internal", "external", or "all" (default: "all")max_links (integer, optional, default: 200) — safety cap on returned linkslinks (array) — list of {url, text, type} objects with resolved absolute URLslink_count (integer) — number of links foundGet page title, description, Open Graph tags, and other metadata. Use for generating previews, indexing, or understanding page context before deeper scraping.
url (string, required) — URL to extract metadata fromtitle (string) — page titledescription (string) — meta descriptionog_tags (dict) — Open Graph metadatacanonical_url (string) — canonical URL if specifiedlanguage (string) — page languageCapture visual page rendering as an image. Use for visual verification, archival, or when page layout matters. Requires a browser-capable environment.
url (string, required) — URL to screenshotwidth (integer, optional, default: 1920) — viewport width in pixelsheight (integer, optional, default: 1080) — viewport height in pixelsfull_page (boolean, optional, default: false) — capture full scrollable heightimage_path (string) — path to saved screenshotdimensions (string) — actual image dimensionsimport urllib.robotparser
def can_scrape(url: str, user_agent: str = "Gorgon-Bot/1.0") -> bool:
"""Check if scraping is allowed by robots.txt."""
from urllib.parse import urlparse
parsed = urlparse(url)
robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt"
rp = urllib.robotparser.RobotFileParser()
rp.set_url(robots_url)
try:
rp.read()
return rp.can_fetch(user_agent, url)
except Exception:
return True # Allow if robots.txt unavailable
import time
import requests
from collections import defaultdict
class RateLimitedFetcher:
def __init__(self, min_delay: float = 1.0):
self.min_delay = min_delay
self.last_request = defaultdict(float)
def fetch(self, url: str) -> requests.Response:
from urllib.parse import urlparse
domain = urlparse(url).netloc
# Enforce rate limit
elapsed = time.time() - self.last_request[domain]
if elapsed < self.min_delay:
time.sleep(self.min_delay - elapsed)
response = requests.get(
url,
headers={"User-Agent": "Gorgon-Bot/1.0"},
timeout=30
)
self.last_request[domain] = time.time()
return response
from bs4 import BeautifulSoup
def extract_tables(html: str) -> list[list[dict]]:
"""Extract all tables from HTML as list of dicts."""
soup = BeautifulSoup(html, "html.parser")
tables = []
for table in soup.find_all("table"):
headers = [th.get_text(strip=True) for th in table.find_all("th")]
rows = []
for tr in table.find_all("tr"):
cells = [td.get_text(strip=True) for td in tr.find_all("td")]
if cells and headers:
rows.append(dict(zip(headers, cells)))
if rows:
tables.append(rows)
return tables
Before reporting scraping results as complete, verify:
Pause and reason explicitly when:
| Error Type | Action | Max Retries |
|---|---|---|
| 403 Forbidden | Respect denial, do not retry | 0 |
| 404 Not Found | Report missing, check URL for typos | 0 |
| 429 Rate Limited | Exponential backoff | 3 |
| Timeout | Retry once with longer timeout (60s) | 1 |
| Encoding error | Try alternative encodings (latin-1, cp1252) | 2 |
| JavaScript-required page | Fall back to Playwright/browser method | 1 |
| Same error after retries | Stop, report what was attempted and failed | — |
If this skill's protocol is violated: