菜单
Extracts, cleans, and structures content from web pages by URL or HTML input
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
| name | web-scraper |
| description | Extracts, cleans, and structures content from web pages by URL or HTML input |
| version | 1.0.0 |
| author | go-on-team |
| tags | ["web","scraping","html","data-extraction","url"] |
| min_go_on_version | 1.0.0 |
Extracts meaningful content from web pages — article text, tables, metadata, links, and structured data — given a URL or raw HTML. Handles pagination, JavaScript-rendered content hints, and common anti-scraping patterns via fallback extraction strategies.
| Parameter | Type | Description |
|---|---|---|
url | string | Target URL to scrape (mutually exclusive with html) |
html | string | Raw HTML to process (mutually exclusive with url) |
extract | string | Content to extract: article, metadata, links, tables, all (default: article) |
max_length | integer | Optional: maximum characters in output (default: 10000) |
include_raw_html | boolean | Optional: include cleaned HTML alongside text (default: false) |
{
"url": "https://example.com/blog/rust-async-await-guide",
"extract": "article",
"max_length": 5000
}
Example output (abbreviated):
{
"url": "https://example.com/blog/rust-async-await-guide",
"title": "A Practical Guide to Rust Async/Await",
"author": "Jane Developer",
"publish_date": "2026-06-15",
"content": "Async/await in Rust provides zero-cost abstractions for concurrent programming...",
"word_count": 1247,
"links": [
{"text": "Tokio tutorial", "href": "https://tokio.rs/tutorial"},
{"text": "Async book", "href": "https://rust-lang.github.io/async-book/"}
]
}
Analyzes project structure, module dependencies, imports, and entry points to generate architecture diagrams in Mermaid format
Analyzes ETL and data pipeline code for optimization opportunities across Python (Pandas, PySpark), Rust (polars, datafusion), SQL, and general pipeline descriptions
Validates environment variable configurations and config files (YAML, TOML, JSON, .env) for missing required variables, type mismatches, deprecated keys, naming convention violations, secret exposure risks, and invalid value ranges
Analyzes code for performance bottlenecks including N+1 queries, O(n^2) or worse algorithms, unnecessary allocations, sync I/O in async contexts, excessive cloning, missing caching opportunities, and large payload transfers. Supports Rust, Python, TypeScript, and Go.
Analyzes, improves, and restructures LLM prompts for clarity, efficiency, and reliability
Analyzes source code for common security vulnerabilities including SQL injection, XSS, command injection, hardcoded secrets, insecure deserialization, path traversal, and SSRF