Menu
Extracts, cleans, and structures content from web pages by URL or HTML input
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Analyzes project structure, module dependencies, imports, and entry points to generate architecture diagrams in Mermaid format
Analyzes ETL and data pipeline code for optimization opportunities across Python (Pandas, PySpark), Rust (polars, datafusion), SQL, and general pipeline descriptions
Validates environment variable configurations and config files (YAML, TOML, JSON, .env) for missing required variables, type mismatches, deprecated keys, naming convention violations, secret exposure risks, and invalid value ranges
Analyzes code for performance bottlenecks including N+1 queries, O(n^2) or worse algorithms, unnecessary allocations, sync I/O in async contexts, excessive cloning, missing caching opportunities, and large payload transfers. Supports Rust, Python, TypeScript, and Go.
Analyzes, improves, and restructures LLM prompts for clarity, efficiency, and reliability
Analyzes source code for common security vulnerabilities including SQL injection, XSS, command injection, hardcoded secrets, insecure deserialization, path traversal, and SSRF
| name | web-scraper |
| description | Extracts, cleans, and structures content from web pages by URL or HTML input |
| version | 1.0.0 |
| author | go-on-team |
| tags | ["web","scraping","html","data-extraction","url"] |
| min_go_on_version | 1.0.0 |
Extracts meaningful content from web pages — article text, tables, metadata, links, and structured data — given a URL or raw HTML. Handles pagination, JavaScript-rendered content hints, and common anti-scraping patterns via fallback extraction strategies.
| Parameter | Type | Description |
|---|---|---|
url | string | Target URL to scrape (mutually exclusive with html) |
html | string | Raw HTML to process (mutually exclusive with url) |
extract | string | Content to extract: article, metadata, links, tables, all (default: article) |
max_length | integer | Optional: maximum characters in output (default: 10000) |
include_raw_html | boolean | Optional: include cleaned HTML alongside text (default: false) |
{
"url": "https://example.com/blog/rust-async-await-guide",
"extract": "article",
"max_length": 5000
}
Example output (abbreviated):
{
"url": "https://example.com/blog/rust-async-await-guide",
"title": "A Practical Guide to Rust Async/Await",
"author": "Jane Developer",
"publish_date": "2026-06-15",
"content": "Async/await in Rust provides zero-cost abstractions for concurrent programming...",
"word_count": 1247,
"links": [
{"text": "Tokio tutorial", "href": "https://tokio.rs/tutorial"},
{"text": "Async book", "href": "https://rust-lang.github.io/async-book/"}
]
}